text:
I recently filled a 3-month RX for dextroamphetamine spansules, 15 mg. This is the first time it was filled by Actavis/Davis and all the capsules are well short of full. This has not happened with other manufacturers and I want to know if anyone else has experienced this? In addition, the medication does not seem to be as effective as others.
Should capsules of dextroamphetamine spansules, 15 mg, be only 3/4 full?
- Asked
- 24 Jan 2015 by Curious2no
- Updated
- 24 January 2015
- Topics
- dextroamphetamine, actavis, capsule
Responses (1)
I have never seen any capsule to be filled to the brim. Actavis is quite a reputable drug manufacturer. If it were anyone else I'd say ok be suspicious but Actavis is going to put in the capsule what is supposed to be in it. Have you ever heard of the placebo effect?
Further Information
- Dextroamphetamine Information for Consumers
- Dextroamphetamine Information for Healthcare Professionals (includes dosage details)
- Side Effects of Dextroamphetamine (detailed)
Search for questions
Still looking for answers? Try searching for what you seek or ask your own question.
Similar Questions
Dextroamphetamine Sustained-Release Capsules?
Posted 20 Aug 2009 • 1 answer
Dexedrine Spansules - I am 41 years old and have ADD. I am currently taking 15mg dextroamphetamine?
Posted 29 Nov 2011 • 1 answer
How full should a amphetamine and dextroamphetamine er capsule be?
Posted 25 Mar 2013 • 3 answers
I am trying to locate Actavis amphetamine and dextroamphetamine?
Posted 26 Jul 2017 • 1 answer
Dextroamphetamine and amphetamine XR capsules the same as amphetamine and dextroamphetamine?
Posted 12 Apr 2018 • 0 answers


text:
MTAA Super is committed to providing excellent service, and any complaints from members are dealt with as soon as possible. If you wish to make a complaint about MTAA Super write to:
The Complaints Officer
MTAA Super
Locked Bag 5134
Parramatta NSW 2124
Your complaint will be dealt with within 90 days. If you believe that your complaint has not been satisfactorily handled or if you do not receive a response within 90 days, you can contact the Superannuation Complaints Tribunal, an independent body established by the Federal Government. The Tribunal has jurisdiction to deal with most types of superannuation related complaints that relate to trustee decisions. The tribunal's contact details are:
Superannuation Complaints Tribunal
Locked Bag 3060
GPO Melbourne VIC 3001
Phone: 1300 884 114
This is a free service.
If your complaint is outside the jurisdiction of the Superannuation Complaints Tribunal and you do not receive a response to your complaint within 45 days, or you are not satisfied with the response, you may be eligible to take your complaint to the Financial Ombudsman Service.
You may contact the Financial Ombudsman Service by calling 1800 367 287 (local call cost in Australia)
This is a free service.
The Australian Securities and Investments Commission has an infoline, 1300 300 630, that you may use to make a complaint and obtain information about your rights.

subset: commoncrawl


text:
There has been lots of hubbub in the last two weeks about California's economy drying up and blowing away like sagebrush after four years of drought. But the nonpartisan Legislative Analyst's Office (LAO), which provides budget advice to state lawmakers, announced this week that "we currently do not expect the drought to have a significant effect on statewide economic activity or state government revenues." The reason: agriculture is only a small piece of the economy.
The LAO report acknowledges that the drought remains a "risk factor" that could have an impact later. However, the current effects on agriculture have not damaged the state's economy as a whole, nor have residential water restrictions.
This economic "good news" from Sacramento is not to say that pain will not be inflicted on certain sectors or individuals in California. Ski resorts, recreational boating sites, and golf courses are going to be hurt by the drought. In response to Governor Jerry Brown's executive order to curtail water use this month, regulators are preparing new restrictions on watering large commercial landscapes, such as golf courses, as well as a prohibition on sprinklers at new homes without water-efficient drip irrigation.
The biggest economic impact of the drought has been felt by rural agricultural communities. The relentless dry spell has already forced farmers to fallow 400,000 acres and driven 17,000 farmhands out of work, according to state officials.
With California's farmers selling almost $50 billion worth of food in 2013, the state made more money off agriculture than any other state in the nation. To accomplish this feat, agriculture consumed about 80 percent of California water usage (once the portion of water that washes out to sea is dropped from the calculation).
Still, the agricultural sector represents only 2.5 percent of the $2 trillion California economy as a whole, and less than 4 percent of Californians work directly on farms and ranches.
Politically, the sector garners a tremendous amount of support from rural communities and California voters have been very empathetic to the plight of agricultural communities.
That explains why California Governor Jerry Brown took the unprecedented step of sparing rural communities by issuing an executive order earlier this month that forced urban water agencies to reduce their water use by 25 percent to save an estimated 1.5 million acre-feet, or nearly 500 billion gallons of water, over the next year. Brown's order exempted most of the state's farmers (though some complain that they were not included).
Tax collections do not seem to be suffering from the drought, with revenue $1.5 billion ahead of projections at the end of March, according to the State Controller's monthly report. Most of the gain is due to California leading the nation with 476,400 new jobs over the last 12 months, on top of capital gains on from Silicon Valley's hot IPO market.
Cyclical droughts and torrential rains have always been part of the California experience. Looking forward, the National Oceanic and Atmospheric Administration (NOAA) on April 9 officially declared a strong El Niño advisory, reflecting substantially above-average surface sea temperatures forming across a huge swath of the equatorial Pacific.
That means that there is a 60 to 70 percent probability that America could experience a monster winter like the El Niño that hit in 1997-1998, causing drenching rains in the Southeast, ice storms in the Northeast, tornadoes in Florida, and mass flooding in California.


text:
Know more about this business than we do? Cool! Please submit any corrections or missing details you may have.Help us make it right
Be the first OregonLive.com user to add photos or videos of Goertz James D DDS
I've been with Dr. Goertz for years and he is excellent and very good at what he does. I would heartily recommend him to anybody seeking a good dentist.
Q: Did this provider thoroughly explain the risks and benefits of your treatment?
A: Yes, they made sure I had all the information I needed to make a decision
Q: Did this provider prominently display their diplomas or business license?
A: Definitely! I saw it as soon as I walked in!
Q: Did this dentist offer you anesthetic when appropriate?
A: Definitely! They paused every few minutes to make...
James D Goertz, DDS practices as a Dentist in Oregon City, OR.


text:
Tyson Hunter dresses sharply, works out almost every day and can't wait to make his mark on the business world.
Hunter, 23, also happens to owe $152,000 in student loans, accumulated in four years at Boston University. He graduated last year with a degree in business administration and now earns $40,000 a year at a market-research company.
His loan payments soon will top $1,000 a month, about a third of his salary. If he makes the minimum payments, he will retire his student debt when he is 53, having handed lenders some $300,000.
"Buying a house? That's not even in the 10-year goals," says Hunter, who has temporarily moved back into his mom's condo in Washington state.
At a time when deep uncertainty permeates the economy, graduates are entering the work force with staggering liabilities. The average student debt has doubled since the mid-1990s.
And that burden often has an effect on the most fundamental choices students are making — decisions about home, family and career.
Take Isiah Sandlin, 32, and Hollie Sexton, 26, who are studying medicine at the University of Washington. Sandlin already has $275,000 in loans; Sexton, $100,000. When the couple graduates in two years, they expect their combined loans will top a half-million dollars. Each new loan helps cover the payments on previous ones.
At least one of them will likely need to work in a high-paying specialty to make the whole thing fly. Sexton's dream of volunteering abroad seems a long way off.
"I couldn't quit now if I wanted to. No way," she says. "Once you are on the train, you've got to keep going."
While Hunter and Sandlin have exceptionally large loans, more than two-thirds of all students now borrow money to finance their education, up from less than half in 1993. Among undergrads who borrow, the average finished school in 2004 with loans of $19,000, up from $9,000 a decade earlier, according to one analysis of federal data.
Educators and economists have argued for decades that higher education represents a great long-term investment, thanks to the higher wages graduates can command. Janet Cantelon, director of student financial services at Seattle University, points out that even $300 a month is manageable for most graduates – the equivalent of a car payment – and a good long-term investment.
Yet the payoff is simply not as good as it once was.
Workers with bachelor's degrees do earn more – an average $51,000 a year, compared with $31,000 a year for high school graduates, according to the U.S. Department of Labor. But the department also reports that college tuition now costs five times what it did in the early 1980s, and it is rising at more than twice the rate of inflation. Inflation-adjusted wages, meanwhile, have remained stagnant since 2002.
Driving up college costs in recent years is the fact that states are investing less in public universities, putting more of the burden on students, says Jacqueline King, an assistant vice president at the Washington, D.C.-based American Council on Education.
With its professors and tutors, administrators and groundskeepers, education is labor-intensive, King says. The cost of employing skilled professors has risen sharply. And universities can't keep costs in check the way big business can, she added, by outsourcing or manufacturing overseas.
To be sure, student loans do help hundreds of thousands of students each year make it through college. And there are signs that parents are coming to grips with the new financial reality of college: Assets in so-called 529 college-savings plans grew from $15 billion in 2001 to $122 billion in 2007, according to the College Board.
But for students graduating now, large loan repayments are adding a significant financial burden at a time when they also face rising health care costs, expensive housing options and a difficult employment market.
Sandlin and Sexton say the amount they owe is a concern.
"I'd be lying to say it didn't color my specialty," said Sandlin, who is looking into emergency medicine or a surgical specialty. "Some of the specialties I'm considering are of interest to me, but they also pay particularly well on the whole physician spectrum."
On Sexton's 2008-09 financial-aid statement, UW outlines the expected cost of medical school: $50,500, including tuition, housing and personal expenses. The university subtracts "total resources" – zero – and offers a package that includes $3,000 in grants and $47,500 in federal loans.
Sexton is taking the full amount. She doesn't feel like she has any other option.


text:
Yes! Here's how it works:
- First we'll need your artwork
- Next, we make a mold based on that artwork for a one-time fee of $3000 and the mold is yours to keep.
- Then we'll use your mold to manufacture your capsules with your logo.
Please note: Custom capsule orders require a minimum order of 1,000,000 capsules.
Contact us for all details and pricing at 1-866-536-CAPS (2277)


text:
MLS newcomers Atlanta United have announced yet another high-profile signing ahead of their maiden season in the top flight.
Using Targeted Allocation Money, the MLS side have acquired the services of Chilean midfielder Carlos Carmona, who joins from Serie A high-flyers Atalanta.
The 29-year-old spent the majority of his professional career in Italy, representing Reggina and Atalanta, making 155 appearances in Italy's elite division.
But, having featured in just two league games this season before picking up a serious injury, Carmona was allowed to embark on a new challenge.
The Chile international is the second footballer to join Atlanta United from Serie A in last seven days, following in the footsteps of former Torino striker Josef Martinez.
The MLS new boys have been quite busy in the current transfer window, adding the likes of Greg Garza, Kenwyne Jones, Tyrone Mears and Jeff Larentowicz among others, while agreeing a summer deal for Brad Guzan.
- Soccer News Like
- Be the first of your friends!


text:
Makeup for brides
Look beautiful on your wedding day is easier than it seems. You just need a good makeup artist and several tests. Discover what you...Read more »
Makeup tips for brides
Makeup should be as natural as possible and uncluttered: makeup tricks and be radiant wedding day. To look like a princess, a beautiful wedding...Read more »


text:
[English | Japanese][Home]
SPring-8 RIKEN Beamlines
RIKEN SPring-8 Center operates 8 beamlines in SPring-8 for study in the field of physics, biology and material science. We promote and endorse RIKEN projects with synchrotron radiation.
BL17SU RIKEN Coherent Soft X-ray Spectroscopy
BL17SU is aimed at advancing the spectroscopic studies for mainly solid state physics and materials science using high brilliant soft X-ray undulator radiation. Photoabsorption, photoemission and soft X-ray emission spectroscopies are adopted to investigate the electronic structure of various kinds of materials. The branched beamlines (a and b), which can be switched by the pre-mirror and used alternatively, are ready for use. In the a-branch, there are four experimental stations (A1, A2, Ac, and A3). In A1 station, the photon-ion merged-beam apparatus is installed on the slide-rail, and that enables us to open the A1 station for free space where users can connect their own instruments. In A2 station, materials science by high resolution photoemission spectroscopy is performed to study the electronic structure of various kinds of materials. In Ac station, there is a small free space where users can carry in their own instruments. In A3 station, the electronic structure of solid as well as biological samples is investigated using soft X-ray emission spectroscopy. In the b-branch, on the other hand, there are two experimental stations (B1 and Bc). In B1 station, the apparatus for the soft x-ray diffraction spectroscopy is installed to investigate the electronic structure of long-range ordered materials. In Bc station, a small free space where users can carry in their own apparatus is available with the use of focused soft x-ray beam. (more)
BL19LXU RIKEN SR Physics
BL19LXU is equipped with a 27-m-long undulator as the brightest X-ray source. The optics hutch contains a Si (111) double-crystal monochromator with cryogenic cooling and a double mirror with the SPring-8 standard mechanism. EH3 is an open hutch, where users can bring in their own apparatus. We will welcome researches in any field, just requiring the highly brilliant X-ray beam. (more)
BL26B1, BL26B2 RIKEN Structural Genomics I & II
BL26B1 & B2 have been developed as RIKEN Structural Genomics Beamline I & II for high throughput protein crystallography (HTPX). The function of the beamline is to collect diffraction data of a large number of protein crystals for structural analysis. In the experimental hutch, an automatic sample changer developed at BL26 is installed to improve the throughput of the experiment. Cooperating with the sample changer, the beamline scheduling software executes the successive data collections without any user intervention. The beamline design is based on the SPring-8 standard optics for the bending magnet, which is simple and remote controllable. Two type X-ray detectors (IP and CCD) can be automatically switched according to the experimental schedule. (more)
BL29XU RIKEN Coherent X-ray Optics
X-ray optics, especially coherent X-ray optics (more)
BL32XU RIKEN Targeted Proteins
Most of proteins substantially important in biological research have difficulties in growing single crystals of sufficient size. Demands for synchrotron radiation facilities for providing innovative technologies to achieve protein crystallography with micro-sized crystals are increasing more and more. At SPring-8, a new protein micro-crystallography beamline BL32XU is under construction for Targeted Proteins Research Program promoted by MEXT of Japan. The beamline is designed to provide the stabilized and brilliant micro-beam to collect high-quality data from micro-crystals. The highly brilliant micro-beam will be a powerful tool to obtain diffraction data with high signal-to-noise ratio even from weakly diffracting small crystals, by enhancing diffraction signal, and also by reducing background scattering around sample. User operation is planned to start in April 2010.
BL43LXU RIKEN Quantum NanoDynamics
BL43LXU, the RIKEN Quantum NanoDynamics Beamline, is dedicated to studying fast dynamics of materials on the nano-scale (i.e. correlations on atomic length scales) using non-resonant inelastic x-ray scattering. This includes both electronic dynamics, and atomic (phonon) dynamics. The beamline will eventually have 2 spectrometers, one dedicated to medium (10-100 meV) resolution work with large momentum acceptance, for weaker (electronic) excitations and one dedicated to high (<1 to 6 meV) resolution for phonon studies and for some classes of electronic excitations. As experiments are very much flux limited, an important aspect of this beamline is that it will optimize the flux onto the sample. For fall of 2016 the high-resolution spectrometer (0.75-2.8 meV resolution) is opened for the users. Before submitting the application for the first time, please contact the beamline scientist in charge. (more)
BL44B2 RIKEN Materials Science
The BL44B2 is a materials science beamline using high-energy powder diffractometry.The main scientific purpose is to reveal correlations between structures and properties in novel functional materials based on the charge density structure. To achieve the purpose, the optical parameters have been optimized and the powder diffractometer has been developed. Taking advantage of the beamline, high counting statistics and high angular resolution data can be collected in several minutes from powder samples with several milligrams. The white beam generated from the SPring-8 standard bending magnet is monochromatized and focused by the optical components. The beam optimized for powder samples sealed in a glass capillary is irradiated to the samples on the diffractometer, by which a camera method with transmission geometry allows to collect a whole diffraction pattern simultaneously. (more)
BL45XU RIKEN Structural Biology I
BL45XU equipped with tandem vertical undulators consists of two experimental stations; small-angle x-ray scattering (SAXS) and small- and wide-angle scattering (SWAXS). Both experiments can be independently carried out. Only SAXS station is opening for the public use. The main characteristic of BL45XU-SAXS is its idealistic optics for scattering experiments from non-crystalline biological samples as well as polymer materials. Double-crystal monochromator (synthetic diamond (111) in Laue-Bragg geometry) followed by K-B mirrors (vertical and horizontal) are used in order to reduce parasitic scattering. The change of camera length in limited beamtime is strongly avoided. We contribute 20% of beam time for public use. As for biological sample we also accept extra external users within the framework of collaboration. Before submitting the application to JASRI for the first time, please talk with contact person. (more)


text:
F.S. Commercial Landscape Inc. Offers Commercial Landscape Services for Alameda County
Commercial landscaping services offer a broad range of solutions for properties throughout Alameda County. From ongoing maintenance to commercial landscape design, F.S. Commercial Landscape Inc. provides affordable solutions and reliable expertise service that will delight customers, residents, employees and visitors alike.
Commercial real estate is a highly competitive industry, with companies vying for renters, residents, owners and customers. With a dynamic commercial landscape design, your property will stand out and make a great first impression.
The right landscape design adds a wow factor and raises eyebrows when prospects first visit. At F.S. Commercial Landscape, we have a talented and award-winning team of designers, horticulturalists and technicians to help create the professional design you expect.
A great commercial landscaping design also adds to the property value, whether it's a retirement or condo complex, apartment building, mall, office building or corporate complex.
We start with an initial consultation with Alameda County customers to understand the needs and desires for the property. We conduct a comprehensive site inspection and analysis in order to create a budget-conscious, manageable and innovative design that's customized for the property.
Our landscape architects and designers emphasize sustainable and native plant materials, green landscape maintenance and scalable concepts that can grow as the site expands.
Once the landscape design plan is approved, our implementation teams work to install the solution quickly and thoroughly. With regular communication with our customers and designers, we take great care to create the ideal solution that will meet all of your expectations and needs.
A commercial landscape design can feature any number of components, such as:
Trees and shrubs whether new or transplanted
Seasonally appropriate flowers and plant beds
Outdoor seating, including benches, gazebos and patios
Other hardscapes such as retaining walls, fencing, pathways and sidewalks
Irrigation, stormwater and drainage systems
Landscape lighting
Our Approach
Your commercial landscape design does far more than create a first impression. At F.S. Commercial Landscape, we focus on solutions that are cost-effective to install and maintain your landscape. We look at each solution as a revitalization project that helps bring a property to life or infuse it with a new look and energy.
Our solutions are also designed to celebrate and announce new management or new emphasis on a property. Our landscaping design focuses on user safety and crime reduction, providing pathways, walkways, open space and lighting solutions that enhance and expand the functional use of the land.
Commercial Landscape Maintenance and Other Services
In addition to our award-winning landscape design services, F.S. Commercial Landscape also provide ongoing maintenance services for your commercial property. Maintenance ensures that landscapes are well maintained, working properly and replaced or repaired in a timely manner when necessary.
Our maintenance and other services include:
Mowing, trimming and edging
Fertilizer and pesticide treatments
Tree trimming and removal
Shrub pruning
Sustainable irrigation system installation and management
Integrated pest management
Landscape conservation analysis
For more information on F.S. Commercial Landscape design and maintenance services, in Alameda County and a free consultation, contact us at (510) 623-1225.
Posted on 02/19/2017 at 10:30 AM
FS Commercial Landscape
43197 Osgood Rd
Fremont, CA
Phone: (510) 623-1225
Email: firstname.lastname@example.com
FS Commercial Landscape is your premier landscape contractor in both Northern and Southern California. We specialize in Commercial Sustainable Landscapes, Landscape Maintenance, Landscape Design, and Tree Services.
San Francisco Bay area office provides services in and around, Livermore, Dublin, Pleasanton, Fremont, Palo Alto, Sunnyvale, Walnut Creek, Oakland, Hayward, South Bay, Los Gatos, Mountain View, Antioch and Pleasant Hill. Our Southern California office is located in Riverside and services Riverside County, Orange County and the Inland Empire


text:
ERIC Number: ED167746
Record Type: RIE
Publication Date: N/A
Pages: 91
Abstractor: N/A
Reference Count: 0
Career Development Partnership. A Program Linking Parents, Students, Schools. A Guide to Implementation.
Myers, Edward J.; And Others
A program was developed for parent involvement in career education of students at the elementary, junior high, and senior high levels. The goal of the program is to make parents aware of the important role they play in the career development processes of their children. Topics dealt with include the following: (1) improvement of communication skills between parent and child; (2) influence of parents on career choice and work values; (3) career resources in the school and community; (4) sex-role stereotyping and its effect; (5) improvement of self-concept of children; (6) the importance of career planning; and (7) the stages of career development. At each of the three academic levels, the parents attend five learning sessions, and in Project Apollo, the senior high component, the students are also present. Except for the individual meetings in Project Apollo to review the students' records, the sessions involve the use of handouts, discussion groups, audiovisual materials, reading materials, or role playing. (Outlines are included for each session giving objectives, materials, procedure, preparation, guidelines, and alternate programs. Appendixes provide copies of handouts and questionnaires.) (ELG)
Descriptors: Career Choice, Career Development, Career Education, Career Guidance, Career Planning, Communication Skills, Conferences, Elementary Secondary Education, Guides, Parent Child Relationship, Parent Influence, Parent Participation, Program Descriptions, Program Development, Resources, Role Models, School Counseling, Self Concept, Sex Stereotypes, Student Records
Publication Type: Reports - Descriptive; Guides - Classroom - Teacher
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Rhode Island State Dept. of Education, Providence. Div. of Vocational-Technical Education.; Office of Career Education (DHEW/OE), Washington, DC.
Authoring Institution: Cranston School Dept., RI.


text:
We specialise in strategy, concept, tech development, production and distribution.
Do you want to know how deep the rabbit hole goes?
360° video is a spherical film recording in which users can experience, take part of, and learn with a complete view of the captured surroundings – as if they were there.
Virtual Reality allows users to step into and become fully immersed in a digitally rendered world. VR creates the possibility for users to explore completely new environments and venture beyond the boundaries of reality.
Augmented Reality enhances the analogue world by using multiple technologies to blend digital elements, enriching interactions and engaging users on new levels.
We strategize vigorously, we craft with passion, we distribute to the most influential outlets and most importantly - we track and measure the outcomes.
Establish the goals and define the solution.
Craft and develop in house with our industry leading team.
Collaborate with the best influencers and media partners.
Track, measure and translate the data.
"Warpin Media had a great commitment and was very engaged in our project. They are professional, quick, responsive and provided a lot of helpful solutions throughout the whole project."
— Hans Hedberg, Sveriges Byggindustrier
Co-founder & CSO
• Sales and influencer expert.
• Hardcore gamer
• Tech evangelist.
• Office Mario Kart champion.
Co-founder & CTO
• Master of tech.
• Reads novels in binary.
"Warpin delivered quality beyond all expectations, within the set time frame and budget. They are truly a bunch of digital magic artists with a constant positive mindset."
— Sofia Flodmark, Digital Project Manager at Vinter
Warpin Media AB
Blasieholmsgatan 2
111 48 Stockholm
To book your lecture about VR and branding on the platform for your company our team, please reach out to inquire about pricing and availability.


text:
Quick Reference
A form of psychotherapy based on the belief that psychological problems are the products of abnormalities in communication between family members. All family members are therefore seen together, when possible, in order to clarify and modify the ways they relate together (see genogram, paradox).
Subjects: Medicine and Health.
Related content in Oxford Index
Reference entries
Users without a subscription are not able to see the full content. Please, subscribe or login to access all content.


text:
|Constraints With Hidden Rows in a Database
|November, 2009
|Konik et al.
|Method and apparatus for generating a web page
|February, 2008
|Identifier technique for communication interchange
|June, 2008
|Cohen et al.
|Dynamic building of applications
|August, 2002
|Menachemi et al.
|System and method for creating and manipulating information containers with dynamic registers
|January, 2006
|Offline behavior analysis for online personalization of value added services
|July, 2003
|Godon et al.
|Method for deep mapping
|November, 2007
|Johnson et al.
|Method and system for leased asset management
|July, 2003
|Thomas et al.
|Web server, Web server having function of Java servlet, and computer readable medium
|June, 2004
|January, 2010
|World wide web document distribution system with user selective accessing of any one of a stored historical sequence of changed versions of a bookmarked web document
|January, 2003
|Ullmann et al.
Embodiments of the present invention relate to optimizing queries in distributed processing systems, and more specifically, to elimination of blocks such as splits in distributed processing systems such as MapReduce systems using the Hadoop Distributed File System (HDFS).
According to one embodiment of the present invention, a method of and computer program product for optimizing queries in distributed processing systems are provided. A query is received. The query includes at least one predicate. The query refers to data. The data includes a plurality of records. Each record comprises a plurality of values in a plurality of attributes. Each record is located in at least one of a plurality of blocks of a distributed file system. Each block has a unique identifier. For each block of the distributed file system, at least one value cluster is determined for an attribute of the plurality of attributes. Each value cluster has a range. The predicate of the query is compared with the at least one value cluster of each block. The query is executed against only those blocks where the predicate is met by at least one value cluster.
According to another embodiment of the present invention, a method of and computer program product for optimizing queries in distributed processing systems are provided. A query is received. The query includes at least one predicate. The query refers to data. The data includes a plurality of records. Each record comprises a plurality of values in a plurality of attributes. Each record is located in at least one of a plurality of blocks of a distributed file system. Each block has a unique identifier. A block count is determined that indicates the number of blocks in which each of the values of the data appear. A record count is determined that indicates the number of instances of each of the values in each of the attributes. Based on the block count, a profit value is determined that is associated with copying each of the values of the data to a materialized view. Based on the record count, a cost value is determined that is associated with copying each of the values of the data to a materialized view. A predetermined number of values is selected such that the profit to cost ratio is maximal for the predetermined number of values. A materialized view is provided that comprises the predetermined number of values. Whether the query is applicable to the materialized view is determined. The query is executed against the materialized view if it is applicable to the materialized view.
According to yet another embodiment of the present invention, a method of and computer program product for optimizing queries in distributed processing systems are provided. A query is received. The query includes a plurality of predicates. The query refers to data. The data includes a plurality of records. Each record comprises a plurality of values in a plurality of attributes. Each record is located in at least one of a plurality of blocks of a distributed file system. Each block has a unique identifier. For each block of the distributed file system, at least one value cluster is determined for an attribute of the plurality of attributes. Each cluster has a range. Each of the plurality of blocks is traversed to determine unique values located in each block. An index is built that maps the unique values to the block identifiers of the blocks in which the unique values appear. A cache is maintained that contains the identifiers of blocks from which results were received as a result of execution of prior queries. For each predicate of the plurality of predicates, a list of blocks is determined by either: comparing the predicate with the at least one value cluster of each block to determine those blocks where the predicate is met by at least one value cluster; comparing the predicate with the index to determine those blocks where there exists a mapping in the index from a predicate value to the identifiers of those blocks; or determining whether there is a cache hit for the predicate and if there is a cache hit, determining those blocks whose identifiers appear in the cache. An execution list of blocks is determined based on the list of blocks for each predicate. The query is executed against those blocks appearing in the execution list of blocks.
FIG. 1 depicts a plurality of value clusters according to an embodiment of the present invention.
FIG. 2a depicts an inverted index according to an embodiment of the present invention.
FIG. 2b depicts a materialized view according to an embodiment of the present invention.
FIG. 3a depicts an inverted index according to an embodiment of the present invention.
FIG. 3b depicts an adaptive cache according to an embodiment of the present invention.
FIG. 4 depicts cache replacement data structures according to an embodiment of the present invention.
FIG. 5 depicts a computational flow according to an embodiment of the present invention.
An increasingly important analytics scenario for Hadoop involves multiple (often ad hoc) grouping and aggregation queries with selection predicates over a slowly-changing dataset. These queries are typically expressed via high-level query languages such as Jaql, Pig, and Hive, and are used either directly for business-intelligence applications or to prepare the data for statistical model building and machine learning. In such scenarios techniques for avoiding access to irrelevant data can dramatically improve query performance. However, record-level indexing only slightly improves overall query performance, because it does not minimize the number of mapper "waves", which is determined by the number of processed splits. Moreover, key-based partitioning requires data reorganization, which is usually impractical in Hadoop settings. Consequently, there remains a need in the art for a redefinition of data access mechanisms. To this end, the present disclosure describes a framework for boosting the efficiency of query processing in Hadoop or other distributed systems by avoiding accesses of data splits that are irrelevant to the query at hand. Using novel techniques involving inverted indexes over splits, domain segmentation, materialized views, and adaptive caching, the system of the present disclosure avoids accessing irrelevant splits even in the face of evolving workloads and data. Application of the disclosed techniques achieves up to 20× cost savings with small to moderate storage overheads.
Application of the techniques of the disclosed subject matter improves the performance of massive-scale analytics in an important and increasingly common user scenario. Specifically, the present disclosure is applicable to warehouse-like exploratory analysis environments in which the Hadoop open source implementation of MapReduce is used together with a higher-level interface such as Jaql, Pig, or Hive to execute multiple (and often ad hoc) grouping and aggregation queries with selection predicates over a slowly changing dataset. The queries are used either directly for business-intelligence applications or to prepare the data for statistical model building and machine learning. The query workload typically evolves in ways that are not known a priori; i.e., certain attributes and attribute values are intensely queried for a while, after which a different set of attributes and attribute values are frequently queried, and so on, but the precise sequence of query patterns is not known in advance.
One example of such a scenario would be where a credit card company requires leading analytics capabilities for predicting credit card fraud, as well as strong data-warehouse capabilities for analyzing information generated from cardholder accounts, merchants, consumer banks, and more. In such a scenario, a distributed processing framework such as Hadoop may be used to perform data analysis of many years of highly-structured credit-card transaction data. The analytics needed for marketing and risk assessment may be highly varied, may involve custom algorithms and many different third-party modules that required exploratory analysis, and may require various statistical summaries. Multiple selective queries may need to be run on the transaction data in order to support the development of new prediction techniques, e.g., as new fraud patterns emerge. Different portions of the data may need to be repeatedly sampled for building, refining, and validating models, and to materialize data representing different time periods, customer segments, and so on, to feed into the models once they are built.
Another example is a wind-power generation company that simulates global wind patterns and produces petabytes of semi-structured and nested data. Such data may be stored in a Hadoop Distributed File System (HDFS) and may be queried by MapReduce. It may be necessary to explore the simulation output by repeatedly running ad hoc selection queries, continuously evaluating and analyzing the results.
As illustrated by these examples, the data to be analyzed may be structured or only semi-structured. By storing the data is HDFS and querying it via MapReduce, the burdensome need to structure the data into rigid schemas prior to analysis is avoided.
There remains a continuing pressure to enhance query processing efficiency. Indeed, evolving infrastructure-as-a-service (IaaS) pricing models require users to pay according to the hardware and energy resources that they use, so there is an increasing need to reduce such costs. The most important strategy for improving query performance is to avoid processing data that is irrelevant to the query. However, record-level indexing and key-based partition elimination result in minimal gains in comparison to their complexity.
The key issue for indexing is that the wall-clock execution time for a query depends on the number of "waves" of mappers that are executed, which in turn depends on the number of processed splits—logical partitions of the data as specified in the Hadoop InputFormat interface. (One mapper task is created for each split.) The cost of processing a split is dominated by the I/O cost of reading the split and the overhead of starting up the mapper that processes the split. So the cost of processing one or two records in a split is typically not much less than the cost of processing every record in the split. A record-level indexing technique that ignores this feature of Hadoop does not perform well. Such techniques do not translate to end-to-end savings unless both the Hadoop Distributed File System (HDFS) and Hadoop itself are thoroughly re-engineered. In contrast, the solutions discussed below do not require such reengineering.
A system in which only clustered indexes on the keys are used to partition the data into splits, do not realize substantial efficiencies in themselves. For example, suppose that the data are partitioned into splits by time and that transactions in each split are sorted by their timestamps. Consider the query that returns all transactions between January 2008 and August 2008 for a given customer at a given store. Using a clustered index on dates, the system can eliminate (i.e., avoid the reading and processing of) splits that do not belong to this time interval. If, however, the customer only shops at the store once per month, many splits may be processed that contain no transactions for that customer. In this case, a secondary index on customer IDs would eliminate many splits. Similarly, if the store opened on Jul. 1, 2008, then a composite index on (store, date) value pairs would be more effective than just the clustered index on dates. Where the workload is generally unknown a priori, there is no guidance on which indexes to construct.
Physically partitioning the data according to some key, and then avoiding processing those partitions not referenced by the query, also has limited effectiveness. In one implementation, an expensive map-reduce job is required to partition the data (unless the data is already partitioned by the key), because HDFS does not provide any direct user mechanisms for data placement. If a good partitioning key is not known in advance—the usual case—then reorganizations may need to occur frequently, adding enormous overhead to query processing.
Further complicating attempts at both indexing and partition elimination is the fact that the nature of data evolution in Hadoop differs from that in a classical database system. Specifically, HDFS stores large files as a series of blocks distributed over a cluster of data nodes and replicates for purposes of fault tolerance. HDFS does not directly support in-place updates or appends to existing files; instead, new data (in the form of files) rolls in and old data is rolled out by deleting or archiving files.
To address these issues, the present disclosure provides a framework for boosting the efficiency of query processing in a distributed file system such as the Hadoop/Jaql processing environment. The present disclosure provides for the elimination of splits using enhanced, split-oriented indexing techniques that are coarser, but more effective, than record-based indexing. Furthermore, the present disclosure provides a flexible indexing scheme, based on the observation that non-key fields having large numbers of distinct values may be very effective for split elimination.
Specifically, the techniques of the present disclosure operate over a "flattened JSON" view of the data in a file in which each file and split consists of arrays of records. Each record is a collection of fields. In some embodiments, a field consists of a (name, atom) pair. In other embodiments, a field consists of a (name, atom*) pair, where arrays of atomic values are handled by creating multiple (name, atom) pairs, one for each atom in the array. An atom is an elementary data value that may be for example, a string, date, boolean, or number. As discussed below, the file of interest need not actually be stored in JSON format. Certain split-level statistics are computed for each field—such as min, max, and ranges for a numerical field. These statistics are stored in a "range index". A split-level inverted index is automatically built over all string fields. All fields are indexed, and not just the partitioning field(s); as indicated above, such comprehensive indexing greatly increases opportunities for split elimination. Moreover, there is no need to guess at a partitioning key or to repeatedly re-partition the data as the workload changes; an index is created on all fields once, and the cost of creating the index is then amortized over multiple queries. A cataloging service may be used to maintain the metadata; this service may be lightweight and inexpensive.
One challenge to the inverted-index scheme is the presence of "nasty" atoms, that is, values that are globally infrequent but appear once or twice in a large number of splits. These values are automatically identified, and the inverted index is enhanced with materialized views over the data records having nasty atoms. Then, given a query predicate that references a nasty atom, only a small number of splits in the materialized view need be accessed, rather than a large number of splits in the original file. A similar problem is posed by conjunctive queries that reference "nasty" atom-pairs, where each atom in the pair appears in many splits, but the atoms appear jointly in only a few records. One example is given by (ship date, delivery date) atom-pairs; a given individual ship date may appear in many transactions, and similarly for a given delivery date, but if these dates are unusually close or far part, the number of actual transactions that contain both dates simultaneously will be small. Substitute products are another example: many people buy iPhones or Droids, but few people buy both together. Because of the prohibitive quadratic number of atom-pairs, a complete index for all atom-pairs is too expensive to compute and store. Hence, a novel main-memory adaptive caching method is employed to maintain an inverted index only of those nasty atom-pairs that are the most "valuable" in that they are referred to frequently and result in the elimination of many splits when cached.
As mentioned above, embodiments of the present disclosure compute range statistics for each split, based on a domain segmentation scheme, and stores them in a range index. The range index allows the system to effectively prune splits that do not satisfy given range predicates in a query of interest. Finally, embodiments of the present disclosure combine the foregoing indexing techniques at query execution time to access the minimal set of splits required to process the query.
Embodiments of the present disclosure avoid the need for user-specified physical design, data movement or reorganization, or a priori knowledge about the query workload (though workload information can be exploited if available). Experiments using data with hierarchical and flexible schemas, as well as data with flat and fixed schemas, demonstrate savings in query response time of up to 20× due to split elimination while requiring a small to moderate storage overhead (5%-21%) for the corresponding indexes and materialized views. The indexing techniques of the present disclosure may additionally be applied in the setting of traditional parallel databases.
According to embodiments of the present disclosure, queries with selection predicates are sped up by avoiding useless splits to the greatest extent possible. Disclosed below are the computation of the range index and the inverted index as well as the identification of nasty atoms and the use of a materialized view to tame them. Also disclosure below is a main-memory adaptive caching algorithm used to handle nasty atom-pairs.
As indicated previously, embodiments of the present disclosure can operate over a wide range of storage formats. In some embodiments, Jaql is used for processing, which allows exploitation of the adapters and converters provided by Jaql for translating the actual storage format into a JSON view for processing. Embodiments of the present disclosure can handle formats including CSV files and HDFS sequence files. In some embodiments, the disclosure techniques operate on JSON views generated by Jaql file descriptors. A file descriptor specifies the specific adapters and converters used for the data, as well as the InputFormat used for creating splits. Embodiments of the present disclosure may generate different indexes on the same data if different file descriptors are used. Therefore, for each file in the warehouse, both a file descriptor and file signature are maintained in the system catalog to ensure that the correct index is used to eliminate splits in a Jaql query over the file. Embodiments of the present disclosure use a flattened JSON view of the data. In general, JSON data comprises atoms, arrays, and records (sets of name-value pairs), which can be arbitrarily nested to any depth. Embodiments of the present disclosure flatten this nested structure so that each field name becomes a root-to-leaf path name and each field value corresponds to the atom(s) at the end of the path.
Referring to FIG. 1, domain segmentation for range indexing is depicted. Values are depicted in a natural order along number line 101. Clusters 102, 103, 104 and 105 each comprise a plurality of values, with no values appearing between clusters. Cluster 102 ranges from a1 to b1. Cluster 103 ranges from a2 to b2. Cluster 104 ranges from a3 to b3. Cluster 105 ranges from a4 to b4.
By creating a range index which contains, for each split, range statistics, splits that do not satisfy a given range or equality predicate appearing in the query of interest are quickly eliminated. E.g., if the minimum and maximum values of "weight" in a given split are known to be 2 lbs and 3 lbs, respectively, then it is known immediately that no records in the split satisfy a predicate of the form "weight between 1.2 lbs and 1.5 lbs". However, if the atoms within a split are clustered in small subranges of the overall min-max range, then merely storing the minimum and maximum values for a field may lead to many false positives. For example, suppose the atoms for a given field in a given split are as in FIG. 1, so that the minimum and maximum atom values are a1 and b4. Then the naïve approach would incorrectly assume that there may be records in the split that satisfy the predicate "value between x1 and x2" if x1>b1 and x2<a2. In some embodiments, range statistics are computed for a subset of fields having a given type. For example, range statistics may be computed for only numerical and date field, or alternatively may be computed for numerical, date, and string fields. In other embodiments, range statistics are computed for all fields, or for a predetermined subset of fields.
Embodiments of the present disclosure therefore uses a fast one-dimensional domain segmentation technique that generates multiple ranges for each field in each split. Given a bound k on the number of ranges for each field, and a set of values {v1, . . . , vn}, the technique computes at most k segments of the min-max interval, i.e., at most k ranges, such that each vi is contained in some range and the ranges are configured "as tightly as possible." Exemplary resulting clusters 102, 103, 104, 105 are depicted in FIG. 1.
The algorithm works as follows for a given split: the distinct values among v1, . . . , vn are determined and sorted. If the number of distinct values is at most k, a separate range [vi, vi] is created for each distinct value. Otherwise, there are l>k distinct values, denoted in ascending order as vmin=a1<a2< . . . <al=vmax. The l−1 consecutive gaps gi=ai+1−ai are computed and sorted in descending order. Let i(1), i(2), . . . , i(k−1) denote the indices of the k−1 largest gaps. For each gap gi(j), the interval (ai(j), ai(j)+1) is removed from [vmin, vmax]. What remains is a collection of k ranges—i.e., subintervals of [vmin, vmax]—of the form [c1, d1], [c2, d2], . . . , [ck, dk], where each ck and each dk is equal to one of the vi's. These are the ranges returned by the algorithm. If a predicate does not hit any of the ranges, the split can be eliminated.
For integer-valued and date-valued fields, ai+1−ai−1 is used rather than ai+1−ai to define the size of the gap gi. The former expression reflects the number of possible predicate values inside the gap that can affect the expressions Peq and Prange. In particular, gaps between two consecutive integers are never selected, because there is no benefit in doing so. If it is found that fewer than k−1 gaps have size gi=ai+1−ai−1>0, the algorithm halts and forms exactly those ranges determined by the positive gaps.
Under a query workload having uniformly distributed equality and range predicates that hit the interval [vmin, vmax], the foregoing scheme produces ranges that maximize the probability that a predicate will eliminate the split. To see this, assume without loss of generality that vmax−vmin=1. First consider a random equality predicate qe(attr, w) that searches for records with attribute attr equal to w. The probability that the split is eliminated equals the probability Peq that w lies in one of the k−1 selected gaps, where Peq=Σj−1k−1 gi(j). Similarly, for a predicate qr(attr, w1, w2) that searches for all records having a value of attr between w1 and w2, the split is eliminated if both w1 and w2 lie in the same gap, which happens with probability Prange=Σj−1k−1 gi(j)2. Because gi(1), . . . , gi(k−1) are selected to be the k−1 largest gaps, it follows that under the condition that there are at most k ranges, the choice of ranges simultaneously maximizes Peq and Prange.
If there happens to be workload information available, it may be exploited by redefining each gi as the fraction of historical predicate values that have fallen in the interval (ai, ai+1); thus gi estimates the probability that a future value will fall in this interval. Both the segmentation algorithm and optimality argument remain essentially unchanged.
Referring to FIG. 2a, an inverted index is depicted. Split-Level inverted index 201 refers to splits 202, 203, 204, 205, 206 of file 207.
Embodiments of the present disclosure build an inverted index 201 over each string field in the data file 207 and uses the index when evaluating equality predicates. The index 201 is implemented using fixed-length bitmaps, where the number of bits in each bitmap equals the number of splits in the dataset. The bitmap for a given atom v has its ith bit set to 1 if and only if the ith data split contains v. To compactly represent the index, each bitmap is compressed using Run-Length Encoding (RLE). RLE compression may reduce the index size by up to 85%.
Embodiments of the present disclosure construct the inverted index 201 by performing a single MapReduce job over the data. The map phase reports each atom v and the split ID in which v appears. The reduce phase then groups these output records according to v and merges the split IDs to form the bitmap array, which is then compressed using RLE.
Referring to FIG. 2b, a materialized view is depicted. Materialized view 208 is constructed from splits 202, 203, 204, 205, 206 of file 207.
FIGS. 2a and 2b give an example of a nasty atom v that appears once or twice in each of the N splits (202, 203, 204, 205, 206) of "File A" 207. If an inverted index is used as in FIG. 2a, all of these splits will be accessed whenever an equality predicate involving v is evaluated. However, by building a materialized view 208 on File A 207 that stores all records containing v, queries involving v can be answered by processing the M splits of the materialized view instead of the N splits of the original data, where M<<N.
The number of nasty atoms in a dataset may be very large and the allotted space for the materialized view may be very small, e.g., 1% or 2% of the data size. Thus it is necessary to choose the "best" subset of nasty atoms to store in the materialized view. This optimization problem is formulated as a submodular knapsack problem and embodiments of the present disclosure provide a practical approximate solution.
Denote by M and R the maximum number of splits and records that can fit in the materialized view. Also let splits(v) and records(v) be the set of splits and set of records containing a given atom v. Denote by |splits(v)| and |records(v)| the cardinalities of these sets. Each atom v that is considered for the materialized view has a profit defined as profit(v)=|splits(v)|−M and representing the number of splits saved due to reading splits in the materialized view instead of the original data splits containing v. For a set V of atoms considered jointly for inclusion in the materialized view, the overall profit is defined as profit(V)=ΣνεV p(ν)·profit(v), where p(v) is the probability that a query will contain an equality predicate that references atom v. The probability distribution p may be estimated from workload data if it is available, thus giving higher weight to frequently referenced atoms. If the query workload is unknown, then a uniform query distribution over all atoms in the dataset is assumed; in this case p(v) can be treated as a constant and effectively ignored in the analysis. In a similar manner, cost(v) is defined as the number of records that will be copied to the materialized view and thus use up part of its allotted space. The number of records may be used as a proxy for space consumed because it is extremely expensive to track the actual space consumption associated with each record. Noting that the sets records(u) and records(v) can overlap for distinct u and v, the cost of storing a subset V of atoms in the materialized view is defined as cost(V)=|∪νεV records(v)|.
The optimization problem is therefore to choose a subset of atoms V to maximize profit(V) subject to the constraint that cost(V)≦R. The function cost(V) is submodular and hence the optimization problem is a special case of the submodular knapsack problem. Unfortunately, this problem is very hard even to approximate: it is known that there is no approximation better than a multiplicative √{square root over (n/log n)} factor. Furthermore, the problem contains as a special case the densest k-subgraph problem, for which no practical approximation algorithms are known. To develop a practical solution for constructing the materialized view, the overlap among record sets is ignored, and the cost is approximated (actually, overestimated) as cost(V)≈ΣνεV cost(v)=ΣνεV |records(v)|. In this case, the submodular knapsack problem reduces to a classical 0−1 knapsack problem. This approach yields good results in practice, especially since the space constraints in the problem are not hard constraints—i.e., using slightly more than (or fewer than) M splits is acceptable.
Even the simplified 0−1 knapsack problem is NP-complete, but efficient approximation algorithms exist. It is known that a (1−ε)-approximation may be found in time polynomial in 1/ε and the number of elements (a fully polynomial-time approximation scheme (FPTAS)). Embodiments of the present disclosure, because they are dealing with massive datasets, use a simple greedy algorithm that provides a 0.5-approximation in the worst case. Where the costs are very small compared to the capacity of the knapsack, the greedy algorithm performs much better. Modifications to this algorithm are presented below to efficiently construct a materialized view over a given dataset in a scalable manner. Pseudocode is provided at Inset 1.
|Inset 1: Filtered-based greedy algorithm for
|selecting the materialized view atoms.
|selectNastyAtoms(S, R, L, α)
|- Sets of n values, S = {v1, ..., vn}, // S contains all values
|in the dataset
|- Maximum number of records in the materialized view R
|- Minimum number of splits a value may appear in L.
|- Overlapping factor α ≧ 1.
|- OutputList = { }
|// Output list containing the
|selected nasty values
|Compute ∪ = R/L. // ∪ is the upper bound for the number of
|needed values
|Build Top∪ list ← Scan the values in S and keep only the top ∪
|values w.r.t. profit(v)/cost(v).
|Build Sorted∪ list ← Sort the values in Top∪ list descendingly
|w.r.t. profit(v)/cost(v).
|Define totRecords = 0.
|For (1 ≦ i ≦ ∪) Loop
|- Let v = Sorted∪(i).
|- Add v to OutputList.
|- totRecords += |records(v)|. //Add the number of records
|containing v to the counter
|- If (totRecords ≧ R * α) Then // If the number of records
|exceeded the materialized
|- Exit the loop.
|// view capacity, then exit the loop.
|Return OutputList
The naive greedy algorithm sorts all atoms in the dataset with respect to decreasing profit(v)/cost(v) ratio. Then it selects atoms from the top of the sorted list and adds their corresponding records to the materialized view until it is full. This naive algorithm needs to be modified because (1) sorting all atoms in the dataset is very expensive especially when performed over the large-scale data manipulated by Hadoop, and (2) building the materialized view incrementally, e.g., by adding the records corresponding to a selected atom in each step, is infeasible in the Hadoop system since HDFS does not support random I/O over the data. To avoid sorting all the atoms in the dataset, we compute an upper bound U on the number of atoms that can possibly fit in the materialized view (see below). This upper bound is typically multiple orders of magnitude smaller than the total number of atoms in the dataset. Given U, a scan is performed over the atoms and the top U atoms with respect to their profit(v)/cost(v) ratio are kept (Step 2 in Inset 1). This step may be performed using a min-heap structure without any sorting. Finally, U atoms are sorted according to decreasing profit(v)/cost(v) ratio (Step 3), and then atoms are selected from the top of the list until the materialized view is full (Steps 5-10). A sorting step is still needed because U is an overestimate, but its cost is significantly lower than that of sorting all the atoms.
The upper bound U is computed as follows. A lower bound L is first introduced that represents the minimum number of splits in which an atom v must appear to be considered as a candidate for the materialized view. That is, if v appears in less than L splits, then its corresponding records are not deemed worthy of being accepted into the materialized view. Typically, L is defined as a multiple of the materialized view size, e.g., the default value for L in some embodiments is 3M; where M is the materialized view size in splits. Because an accepted atom v will appear at least once in each of L splits, it follows that v will contribute at least L records to the materialized view. The materialized view can hold at most R records, and thus the upper bound on the number of atoms sufficient to fill up the materialized view is U=R/L (Step 1 in Inset 1).
Because the materialized view cannot be built incrementally, the modified greedy algorithm reports the list of chosen atoms that will contribute to the materialized view (Step 7 in Inset 1). As each atom v in the sorted list is added to the materialized view, a counter is incremented by |records(v)|; the counter represents the amount of allotted space used up so far. In Step 9, the algorithm multiplies the maximum capacity of the materialized view (R) by an overlapping factor α, where α≧1, to heuristically account for the possible overlapping among record sets. The overlapping among record sets does not significantly affect the space utilization of the materialized view; e.g., when α is set to 1 (thus ignoring overlaps), the materialized view space is still approximately 95% utilized.
To create the materialized view for a given dataset, a map-reduce job is executed followed by a map-only job. In the map-reduce job, the map phase reports each atom v along with the split ID in which v appears and the number of records containing v in that split. The reduce phase groups the output records according to v, computes the total number of splits and records in which v appears, i.e., |splits(v)| and |records(v)|, and then executes the greedy algorithm of Inset 1. The map-only job then scans the data and copies the records containing the chosen atoms to the materialized view.
FIGS. 3a and 3b illustrate adaptive caching according to embodiments of the present disclosure. Adaptive cache 301 contains data from splits 202, 203, 204, 205, 206 of file 207.
Embodiments of the present disclosure address the issue of nasty atom-pairs in which the individual atoms are frequent and appear in most (or all) of the splits, but the atom-pairs appear jointly in very few records, and hence in very few splits. This situation differs from the one handled by the materialized view described above, which helps only with atoms that appear in many splits but infrequently overall. In the case that discussed below, individual atoms might well appear frequently and hence not appear in the materialized view.
For example, FIGS. 3a and 3b show two atoms v and w such that each atom is frequent and appears in all splits, but the atom pair (v, w) appears in only one split (Split 3). Using the inverted index described above to find the splits containing (v, w) by computing splits(v) and splits(w) and then intersecting these two sets—which is the only strategy available in the absence of any other information—will result in most of the returned splits being false positives. According to embodiments of the present disclosure, techniques are provided for processing queries, denoted by Q(v, w), that return all records containing both u and w; this query will appear as a subquery of any Jaql query with a conjunctive selection predicate that references both v and w. Embodiments of the present disclosure process conjunctive queries involving more than two atoms by processing the atoms in a pairwise fashion and then intersecting the resulting sets of splits. An atom-pair (v, w) is defined to be a nasty atom-pair if S(v, w)>θ, where S(v, w)=|splits(v) ∩ splits(w)|−|splits(v, w)|. Here splits(v, w) denotes the set of those splits that contain the pair (v, w) in at least one record and θ is a (large) user-defined threshold. The quantity S(v, w) is precisely the number of false positives obtained when split lists for individual atoms v and w are intersected in an attempt to find those splits that contain the pair (v, w). Equivalently, S(v, w) is the potential savings in split accesses if the pair (v, w) is cached.
Building composite or multi-dimensional indexes, where the attribute combinations to be indexed are typically user-defined and are derived from workload and schema information are not feasible for a Hadoop scenario, because the query workload and (possibly loose) schema are typically unknown a priori—indeed, users may not even know the fields in the dataset beforehand. Moreover, algorithms for exploring the space of all possible atom-pairs to index are prohibitively expensive in large-scale data sets because they are inherently quadratic in time and space. Moreover, sampling and sketching techniques may be ineffective in finding nasty pairs because pairs are infrequent.
The present disclosure provides an adaptive main-memory caching technique in which the system monitors the query workload and the atom-pairs that are being queried, and then caches the pairs that are (1) recently queried, (2) frequently queried, and (3) nasty in that S(v, w)>θ. As illustrated in FIG. 3b, the cache maintains atom pairs along with a bitmap specifying the splits containing this pair. As shown in the figure, the cache is significantly more efficient than the inverted index in answering query Q(v, w).
In order to monitor the query workload and to compute the actual splits containing a queried atom pair, i.e., to compute splits(v, w) for pair (v, w), Hadoop may be extended by enabling the map tasks to report back a new boolean flag—in addition to the existing system-maintained counters and time statistics—that indicates whether the map task produced any output tuples (True) or not (False). Given this extension, the system performs the steps in Inset 2 to execute a given conjunctive query Q(v, w) and to decide whether or not (v, w) is candidate for caching.
|Inset 2: Executing a query Q(v; w).
|splits(v) ← Probe the inverted index by value v to get its set of split
|splits(w) ← Probe the inverted index by value w to get its set of split
|splitsToQuery(v,w) ← splits(v) ∩ splits(w) // The set of intersected
|Start the query (job) execution over the set of splits in
|splits(v,w) ← Identify the split Ids assigned to map tasks that
|produced output
|S(v,w) = |splitsToQuery(v,w)| − |splits(v,w)| // Number of
|false-positive splits
|If (S(v,w) > θ) Then
|- insertCache((v,w), splits(v,w), S(v,w) ) //
|Call the caching algorithm to insert (v,w)
By monitoring the map tasks that produce output, the systems determines the ID of every split that contains the (v, w) pair in at least one record (Step 5 in Inset 2). If the number of the false-positive splits S(v, w) is greater than the user-defined threshold, then (v, w) is considered for caching by calling the insertCache( ) algorithm presented below (Steps 6-8). Note that these steps are performed only if an initial probe of the cache does not return a split list for the pair (v, w), because the pair is not in the cache. The algorithm probeCache(v, w) used for this initial probe is described below.
As queries are issued over time, more and more candidate pairs are considered for caching. Because the cache size is limited, an efficient replacement policy is needed to keep the most promising pairs in the cache. Although an LRU replacement policy is easy to implement and maintain, LRU does not perform well for many query workloads. The main reason for this poor performance is that LRU takes into account only how recently an atom-pair has been queried, ignoring other important factors such as the potential savings in splits if the pair were to be cached. Therefore embodiments of the present disclosure adopt the SFR (Savings-Frequency-Recency) cache replacement policy, described below.
The SFR policy maintains in the cache the atom-pairs having the highest "weights," where the weight for a pair (v, w) depends on (1) the potential savings in splits due to caching the pair, i.e., S(v, w), (2) the historical frequency with which (v, w) has been queried, (3) how recently (v, w) has been queried. By taking the savings factor into account, SFR is able catch the "precious" atom pairs having high savings and cache them for a while even if they are not queried extremely frequently. Whenever these pairs are queried, the savings in splits is so significant that the overall efficiency of the cache will benefit by reserving some space for these precious pairs.
SFR maintains two data structures as shown in FIG. 4, queryHistory 401 and cache 402. The queryHistory structure 401 maintains a log of queried atom-pairs 403. Stored along with each pair is a partial weight 404, called the FR-score, that reflects both the frequency and recency of queries involving the pair. The FR-score does not take the potential savings into account. A small subset of these pairs is also stored in the cache structure; any pair stored in the cache structure also appears in queryHistory. Embodiments of the present disclosure do not allow the queryHistory table to grow indefinitely. The system allows entries with FR-scores below a user-defined threshold to be periodically pruned, and a timestamp can be used to eliminate entries that have not been updated for a long period of time.
The cache structure 402 maintains a list of nasty pairs. The final weight 405—which combines the FR-score and the potential savings in split accesses—is stored for each pair 406, along with a bitmap identifying the splits that contain the pair. The final weight for a pair (v, w) is obtained by multiplying the FR-score for (v, w) by the savings factor S(v, w).
Inset 3 displays pseudocode for the cache insertion algorithm, insertCache( ), for inserting a new (v, w) pair. Conceptually, with the arrival of each new pair, the FR-score of all items maintained in queryHistory decays in order to indicate that these items are getting older and to put more emphasis on the new pairs. Such decay could be implemented by multiplying the FR-score of every pair in queryHistory by a decay factor (having a value less than 1) every time a new atompair arrives. Because it is very expensive to perform this operation every time a pair arrives, we implement the same idea equivalently by applying a recency factor ρ to the new pair while keeping the other items in queryHistory unchanged. With every new pair, the recency factor gets amplified by a factor of λ>1 (Step 1). Then the algorithm updates the information in queryHistory by increasing the FR-score FR(v, w) by ρ if the pair already exists, or inserting a new record if the pair does not exist (Steps 2-5). The addition operation in Step 3 ultimately gives higher weights to frequently queried pairs. To insert (v, w) into the cache, the final weight W(v, w) is first computed (Step 6). Next, a new entry containing information for (v, w) is inserted into the cache if the cache has available space (Step 8); otherwise the algorithm finds the entry with the smallest weight, say W(v′, w′), and replaces this entry with the new one if W(v, w)>W(v′, w′) (Steps 10-12). Because the recency factor ρ gets amplified with every new pair, it will eventually overflow. To avoid this problem, the system periodically—e.g., after every k new arrivals—normalizes all the scores while preserving their relative orders.
|Inset 3: Inserting into the cache: insertCache( ) algorithm.
|insertCache ((v,w), splits(v,w), S(v,w))
|- (v,w): The value pair to cache
|- splits(v,w): Set of splits containing (v,w)
|- S(v,w): The number of splits when caching (v,w)
|Output: None
|Global Parametrs:
|- Recency factor ρ = 1
|- Amplification factor λ = 1.001 (λ > 1 )
|(1) Increase ρ = ρ x λ
|// Amplify the recency factor
|// Update the queryHistory information
|(2) If (v,w) is in queryHistory Then
|- FR(V:W) += p
|// The addition op. gives higher weight
|for frequent pairs
|(4) Else
|- Insert ((v,w), FR(v,w) = ρ) into queryHistory
|// Insert into the cache
|(6) W(v,w) = FR(v,w) * S(v,w)
|//Compute the final weight of (v,w)
|(7) If cache is not full Then
|- Insert ((v,w), W(v,w), splits(v,w))
|(9) Else
|- W(v′,w′)← Find the entry with the smallest weight, say for
|- If (W(v′,w′) < W(v,w)) Then
|- Replace the entry of (v′,w′) with the entry of ((v,w), W(v,w),
Pseudocode for the probeCache( ) algorithm mentioned previously is given in Inset 4 and is a simplified version of the insertCache( ) algorithm. If the cache is probed with a given pair (v, w) and the pair is in the cache (a cache hit), then the algorithm updates the FR(v, w) and W(v, w) scores in queryHistory and cache, respectively, and returns a list of split IDs corresponding to that pair, i.e., splits(v, w) (Steps 2-5). Notice that if (v, w) is in the cache, then it is guaranteed to have a corresponding record in the queryHistory table. If the probe results in a cache miss, then the algorithm returns Null (Step 7).
|Inset 4: Probing the cache: probeCache( ) algorithm.
|probeCache ((v,w))
|- (v,w): The probing value pair
|- List of splits to query (if cache hit) or Null (if cache miss)
|Global Parametrs:
|- Recency factor ρ = 1
|- Amplification factor λ = 1.001 (λ > 1)
|(1) If (v,w) is cache hit Then
|- Increase ρ = ρ x λ
|// Increase the recency factor
|- FR(v:w) += ρ
|// Update FR-score in queryHistory
|- W(v,w) = FR(v,w) * S(v,w)
|// Update the weight in the cache
|- Return splits(v,w) from the cache
|(6) Else
|- Return Null
SFR is sensitive to the amplification factor λ. The higher the value of λ, the more emphasis is added to the recency factor against the frequency and savings factors (and vice versa). For example, as λ gets larger, the performance of SFR gets closer to that of LRU.
FIG. 5 depicts a computational flow according to embodiments of the present disclosure. To efficiently build the ranges, indexes, and materialized view without unnecessary scans over the data, embodiments of the present disclosure share the map and reduce tasks whenever possible among the different computations. In FIG. 5, a flow diagram of computations according to embodiments of the present disclosure is provided. These computations require only two passes over the data. In the first pass (lower half of the diagram), a map-reduce job 501 is performed in which the map phase 502 computes the range statistics 503 and reports each atom v along with its split ID and the number of records in the split that contain v. The reduce phase 504 groups the records generated during the map phase according to v, builds the full inverted index 505, and applies the modified greedy algorithm in Inset 1 to identify the subset of nasty atoms to be included in the materialized view. In the second pass (upper half of the diagram), a map-only job 506 is executed that copies to the materialized view 507 every data record that contains one or more of the selected nasty atoms. Importantly, any of the indexing components can be turned on or off using boolean flags, depending on user requirements. For example, if the data will not be queried frequently enough, then users may not be interested in paying the cost of building the materialized view. In this case, the analysis tool automatically bypasses the execution of the modified greedy algorithm in the reduce phase and also skips the second map-only job.
Embodiments of the present disclosure apply several techniques for split elimination: range and inverted index, materialized view, and adaptive cache. As described below, these components are used to compute a minimal list of splits to be processed for a query with selection predicates.
The indexes and the statistical information computed for each split may be stored using several methods. They may be stored in a central repository or distributed and stored together with their corresponding splits. As discussed above, Hadoop has significant overheads for creating map tasks. To avoid such overheads, it is desirable to eliminate the splits before the map tasks are created. Therefore, in embodiments of the present disclosure, the range and inverted index are stored in a central repository that is queried once by the Hadoop InputFormat process before Hadoop's JobTracker creates the required mappers and assigns splits to each mapper. In embodiments of the present disclosure an RDBMS is used for this purpose. However, any, possibly lightweight, repository may be used that supports indexes. The materialized view contains copies of some selected records and is stored in HDFS in the same format as the original data file.
An algorithm is provided that takes as input a conjunctive predicate of the form P=p1̂ p2̂ . . . ̂pn and a file name, and returns a list containing the IDs of the splits in the file that must be processed in order to evaluate the predicate P. This algorithm is invoked from the InputFormat, and as a result can be used with any high-level query language on Hadoop, including Hive, and Pig, as well as incorporated into plain map reduce jobs. Both the materialized view and the list of splits in the materialized view are denoted MV. The algorithm first initializes List, the list of split IDs, to contain the IDs of every split in the file, thus representing the set of splits that must be processed in the absence of additional information. The algorithm uses the range and inverted indexes (inverted index for equality predicates and range index for inequality predicates), and the adaptive cache for equality predicates and uses the range statistics for non-equality predicates. (Here = and in are considered to be the equality predicates.) The final list of splits to process is computed as the intersection of the split lists for all of the pi's, since predicate P is a conjunction. Finally, if any one of the equality predicates in P can be applied using the materialized view MV, then the size of the final split list is compared with the number of splits in MV, and the option with the smaller number of splits is chosen.
In the general case where the query contains both conjunctions and disjunctions, the query predicate is converted into disjunctive normal form, the above algorithm is executed on each of the conjuncts, and finally the union of the resulting split lists is taken to compute the final list of split IDs to process. Some care must be taken, however, in the presence of the materialized view. Specifically, if the algorithm returns split IDs from MV for at least one conjunct D then it is possible to access a given record twice, once from MV when processing D and once from the original data file when processing another conjunct Dj for which MV is not used. To avoid this problem, the previous algorithm is modified to return for each conjunct D the split list that is computed without MV usage, denoted by List(Di), and a boolean, denoted by useMV(Di), that indicates whether or not MV could possibly be used to process Di. The final algorithm is provided in Inset 5. It returns the split IDs in MV if and only if MV can be used for all of the disjuncts (Steps 6-7); otherwise, the algorithm returns the union of the split lists for all disjuncts (Step 8).
|Inset 5: Split list computation with disjunctions.
|evaluatePredicate (file, P)
|Input: P = a predicate with conjunctions and disjunctions
|file = the file to query
|Output: List of split IDs to process
|(1) If P contains at least one disjunction Then
|convert P into disjunctive normal form: D1 V D2 V . . . V Dn
|(3) Else D1 = P
|(4) For each Di Loop
|{List(Di), useMV(Di)} = evaluateConjunct′(file, Di)
|(6) If useMV(Di) = true for all 1 ≦ i ≦ n Then
|Return MV
|(8) Else Return ∪1≦i≦n List(Di)
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.


text:
Three common species of elderberry plants grow up to 30 different varieties. All elderberries belong to the Sambucus genus, and the three most common species are the European or black elderberry (S. nigra), the American or sweet elderberry (S. canadensis) and the red elderberry (S. racemosa). The fruits of the red elderberry bush (S. racemosa) are toxic, so it's not commonly grown or planted in the home landscape. Gardeners often use elderberries to make wines, jams, syrups and pies, as the berries offer a sweet and rich flavor when cooked. Choose an elderberry based on the climate, fruit quality and desired physical characteristics of the plant.
Choose an elderberry bush based on your region's climate. The European or black elderberry (S. nigra) grows best in slightly warmer climates, where winter temperatures rarely get as cold as -20 to -25 degrees F. The American elderberry (S. canadensis) can grow in colder regions, where winter temperatures dip down to -40 degrees F.
Determine your desired berry quality and flavor. The black elderberry bush produces berries most commonly used in commercial recipes, syrups and extracts. The American elderberry yields slightly sweeter fruit than the European species.
Choose an elderberry bush based on its mature size and how appropriate it will be for the space in your landscape. The American elderberry usually grows up to 12 feet tall at maturity, while the European species grows to 30 feet in height.
Select an elderberry flower color. Nearly all American elderberry varieties have large, creamy-white flower clusters, while most European elderberries have pink flowers that can grow up to 10 inches across.
Select an elderberry variety by its foliage. European elderberries have dark-purple to purplish-black leaves, while American elderberries have bright to dark green foliage.


text:
The Janashakthi Professional Indemnity Insurance cover can be customised to provide financial protection to professionals against legal liability to their clients for death or bodily injury, or financial loss arising from the neglect of professional duties.
This policy provides indemnity against any claim in respect to damages for breach of professional duty which may be made against a professional due to any negligent act, error or omission alleged to have been committed by them in the discharge of duties by or on behalf of the insured.
Lawyers/Law firms, Doctors/Hospitals, Engineers/Contracting firms, Architects/Architectural firms, Accountancy firms and Travel companies are identified as high risk groups exposed to risks of this nature.
If you need to meet one of our Professional Advisors to tailor make our Professional Indemnity Plan according to the distinct requirements of your organization please click here to Contact Us.Download the sign up form here
Fax: +94 112 132 299
Email: email@example.com
"The Janashakthi team is strong. This is because of an open door policy that thrives on simple, approachable management."
"With Janashakthi you travel beyond borders. I meet new people, we share ideas and I get to build a global network I can tap into with a simple phone call."
"Janashakthi rekindled my passion for this industry,just as it has lit up the lives of a million hearts and homes"