text:
Are you sure?
This action might not be possible to undo. Are you sure you want to continue?
Real World Web: Performance & Scalability
Ask Bjørn Hansen Develooper LLC
April 14, 2008 – r17
• I'm Ask Bjørn Hansen
perl.org, ~10 years of mod_perl app development, mysql and scalability consulting YellowBot
• I hate tutorials! • Let's do 3 hours of 5 minute° lightning talks!
Actual number of minutes may vary
Construction Ahead!
• • • •
Conflicting advice ahead Not everything here is applicable to everything Ways to "think scalable" rather than be-all-end-all solutions Don't prematurely optimize!
(just don't be too stupid with the "we'll fix it later" stuff)
Questions ...
• • • • • • • •
How many ... ... are using PHP? Python? Python? Java? Ruby? C? 3.23? 4.0? 4.1? 5.0? 5.1? 6.x? MyISAM? InnoDB? Other? Are primarily "programmers" vs "DBAs" Replication? Cluster? Partitioning? Enterprise? Community? PostgreSQL? Oracle? SQL Server? Other?
Seen this talk before?
Slide count 200
• •
No, you haven't. :-)
~266 people * 3 hours = half a work year!
0 2001
Question Policy!
• • • •
Do we have time for questions? Yes! (probably) Quick questions anytime Long questions after
Slides per minute 1.75
or on the list!
0.25 2001 2002 2004 2005 2006
(answer to anything is likely "it depends" or "let's talk about it after / send me an email")
2007 2008
The first, last and only lesson:
Think Horizontal!
• Everything in your architecture, not just the front end web servers • Micro optimizations and other implementation details –– Bzzzzt! Boring!
(blah blah blah, we'll get to the cool stuff in a moment!)
Benchmarking techniques
Scalability isn't the same as processing time
• •
• • •
Not "how fast" but "how many" Test "force", not speed. Think amps, not voltage Test scalability, not just "performance"
Test with "slow clients"
Use a realistic load Testing "how fast" is ok when optimizing implementation details (code snippets, sql queries, server settings)
Vertical scaling
• • • • •
"Get a bigger server" "Use faster CPUs" Can only help so much (with bad scale/$ value) A server twice as fast is more than twice as expensive Super computers are horizontally scaled!
Horizontal scaling
• • •
"Just add another box" (or another thousand or ...) Good to great ...
• •
Implementation, scale your system a few times Architecture, scale dozens or hundreds of times
Get the big picture right first, do micro optimizations later
Scalable Application Servers
Don't paint yourself into a corner from the start
Run Many of Them
• • • •
Avoid having The Server for anything Everything should (be able to) run on any number of boxes Don't replace a server, add a server Support boxes with different capacities
Stateless vs Stateful
• • •
"Shared Nothing" Don't keep state within the application server
(or at least be Really Careful)
Do you use PHP, mod_perl, mod_...
• •
Anything that's more than one process You get that for free! (usually)
"The key to be stateless" or "What goes where"
No Local Storage
• • •
Ever! Not even as a quick hack. Storing session (or other state information) "on the server" doesn't work. "But my load balancer can do 'sticky sessions'"
• •
Uneven scaling – waste of resources
(and unreliable, too!)
The web isn't "session based", it's one short request after another – deal with it
Evil Session
Cookie: session_id =12345
Web/application server with local Session store
What's wrong with this?
... 12345 => { user => { username => 'joe', email => 'firstname.lastname@example.com', id => 987, }, shopping_cart => { ... }, last_viewed_items => { ... }, background_color => 'blue', }, 12346 => { ... }, ....
Evil Session
Cookie: session_id =12345
Easy to guess cookie id
Saving state on one server!
Web/application server with local Session store
... 12345 => { user => { username => 'joe', email => 'email@example.com', id => 987, }, shopping_cart => { ... }, last_viewed_items => { ... }, background_color => 'blue', }, 12346 => { ... }, ....
Duplicate data from a DB table Big blob of junk! What's wrong with this?
Cookie: sid=seh568fzkj5k09z; user=987-65abc; bg_color=blue; cart=...;
Good Session!
Web/application server
• Stateless
Users 987 => { username => 'joe', email => 'firstname.lastname@example.com', }, ... Shopping Carts ...
web server! database
memcached cache
seh568fzkj5k09z => { last_viewed_items => {...}, ... other "junk" ... }, ....
• Important data in • Individual
expiration on session objects in cookies
• Small data items
Safe cookies
• •
Worried about manipulated cookies? Use checksums and timestamps to validate
• •
cookie=1/value/1123157440/ABCD1234 cookie=$cookie_format_version /$value/$timestamp /$checksum
function cookie_checksum { md5_hex( $secret + $time + value ); }
Safe cookies
Want fewer cookies? Combine them:
• •
cookie=1/user::987/cart::943/ts::1123.../EFGH9876 cookie=$cookie_format_version /$key::$value[/$key::$value] /ts::$timestamp /$md5
Encrypt cookies if you must (rarely worth the trouble
and CPU cycles)
I did everything – it's still slow!
• • • •
Optimizations and good micro-practices are necessary, of course But don't confuse what is what! Know when you are optimizing Know when you need to step back and rethink "the big picture"
How to not do all that work again and again and again...
Cache hit-ratios
• • • • •
Start with things you hit all the time Look at web server and database logs Don't cache if you'll need more effort writing to the cache than you save Do cache if it'll help you when that one single page gets a million hits in a few hours (one out of two hundred
thousand pages on the digg frontpage)
Measure! Don't assume – check!
Generate Static Pages
• • • • •
Ultimate Performance: Make all pages static Generate them from templates nightly or when updated Doesn't work well if you have millions of pages or page variations Temporarily make a page static if the servers are crumbling from one particular page being busy Generate your front page as a static file every N minutes
Cache full pages
(or responses if it's an API)
• • • •
Cache full output in the application Include cookies etc. in the "cache key" Fine tuned application level control The most flexible
• •
"use cache when this, not when that"
(anonymous users get cached page, registered users get a generated page)
Use regular expressions to insert customized content into the cached page
Cache full pages 2
• • • •
Front end cache (Squid, Varnish, mod_cache) stores generated content
Set Expires/Cache-Control header to control cache times
or Rewrite rule to generate page if the cached file doesn't exist (this is what Rails does or did...) – only scales to one server
RewriteCond %{REQUEST_FILENAME} !-s RewriteCond %{REQUEST_FILENAME}/index.html !-s RewriteRule (^/.*) /dynamic_handler/$1 [PT]
Still doesn't work for dynamic content per user ("6 items in your cart") Works for caching "dynamic" images ... on one server
Cache partial pages
• • • •
Pre-generate static page "snippets"
(this is what my.yahoo.com does or used to do...)
Have the handler just assemble pieces ready to go
Cache little page snippets (say the sidebar) Be careful, easy to spend more time managing the cache snippets than you save! "Regexp" dynamic content into an otherwise cached page
Cache data
• • • • •
Cache data that's slow to query, fetch or calculate Generate page from the cached data Use the same data to generate API responses! Moves load to cache servers
(For better or worse)
Good for slow data used across many pages
("todays bestsellers in $category")
Caching Tools
Where to put the cache data ...
A couple of bad ideas
Don't do this!
• • •
Process memory ($cache{foo})
• • • • •
Not shared! Limited to one machine (likewise for a file system cache) Some implementations are really fast Flushed on each update Nice if it helps; don't depend on it
Shared memory? Local file system?
MySQL query cache
• • • • •
Write into one or more cache tables id is the "cache key" type is the "namespace"
MySQL cache table
metadata for things like headers for cached http responses purge_key to make it easier to delete data from the cache
CREATE TABLE `combust_cache` ( `id` varchar(64) NOT NULL, `type` varchar(20) NOT NULL default '', `created` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `purge_key` varchar(16) default NULL, `data` mediumblob NOT NULL, `metadata` mediumblob, `serialized` tinyint(1) NOT NULL default '0', `expire` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`id`,`type`), KEY `expire_idx` (`expire`), KEY `purge_idx` (`purge_key`) ) ENGINE=InnoDB
MySQL Cache Fails
• • •
Scaling and availability issues
• •
How do you load balance? How do you deal with a cache box going away?
Partition the cache to spread the write load Use Spread to write to the cache and distribute configuration
General theme: Don't write directly to the DB
MySQL Cache Scales
• • • • • •
Persistence Most of the usual "scale the database" tricks apply Partitioning Master-Master replication for availability .... more on those things in a moment Put metadata in memcached for partitioning and failover information
• • • • •
LiveJournal's distributed caching system
(used practically everywhere!)
Memory based – memory is cheap! Linux 2.6 (epoll) or FreeBSD (kqueue)
Low overhead for many many connections
Run it on boxes with free memory ... or a dedicated cluster: Facebook has more than five hundred dedicated memcached servers (a lot of memory!)
more memcached
• • • • • •
No "master" – fully distributed Simple lightweight protocol (binary protocol coming) Scaling and high-availability is "built-in" Servers are dumb – clients calculate which server to use based on the cache key Clients in perl, java, php, python, ruby, ... New C client library, libmemcached
How to use memcached
• • •
It's a cache, not a database Store data safely somewhere else Pass-through cache (id = session_id or whatever): Read
$data = memcached_fetch( $id ); return $data if $data $data = db_fetch( $id ); memcached_store( $id, $data ); return $data;
db_store( $id, $data ); memcached_store( $id, $data );
Client Side Replication
• • • •
memcached is a cache - the data might "get lost" What if a cache miss is Really Expensive? Store all writes to several memcached servers Client libraries are starting to support this natively
Store complex data
• • • •
Most (all?) client libraries support complex data structures A bit flag in memcached marks the data as "serialized" (another bit for "gzip") All this happens on the client side – memcached just stores a bunch of bytes Future: Store data in JSON? Interoperability between languages!
Store complex data 2
• •
Primary key lookups are probably not worth caching Store things that are expensive to figure out!
function get_slow_summary_data($id) { $data = memcached_fetch( $id ); return $data if $data $data = do_complicated_query( $id ); memcached_store( $id, $data ); return $data; }
Cache invalidation
• • • • • • •
Writing to the cache on updates is hard! Caching is a trade-off You trade "fresh" for "fast" Decide how "fresh" is required and deal with it! Explicit deletes if you can figure out what to delete Add a "generation" / timestamp / whatever to the cache key select id, unix_timestamp(modified_on) as ts from users where username = 'ask'; memcached_fetch( "user_friend_updates; $id; $ts" )
Caching is a trade-off
• •
Can't live with it? Make the primary data-source faster or data-store scale!
Database scaling
How to avoid buying that gazillion dollar Sun box
~$4,000,000 Vertical
~$3,200 ( = 1230 for $4.0M!)
Be Simple
Use MySQL!
• • •
It's fast and it's easy to manage and tune Easy to setup development environments Other DBs can be faster at certain complex queries but are harder to tune – and MySQL is catching up!
• •
Avoid making your schema too complicated Ignore some of the upcoming advice until you REALLY need it!
(even the part about not scaling your DB "up")
PostgreSQL is fast too :-)
More data more places! Share the love load
Basic Replication
• • •
Good Great for read intensive applications Write to one master Read from many slaves
reads webservers writes
Lots more details in "High Performance MySQL"
old, but until MySQL 6 the replication concepts are the same
Relay slave replication
• • • •
Running out of bandwidth on the master? Replicating to multiple data centers? A "replication slave" can be master to other slaves Almost any possible replication scenario can be setup (circular, star replication, ...)
webservers writes
data loading script
writes relay slave A relay slave B
slave slave
slave slave
slave slave
Replication Scaling – Reads
• •
Reading scales well with replication Great for (mostly) read-only applications
One server Two servers
reads reads writes writes reads writes
(thanks to Brad Fitzpatrick!)
Replication Scaling – Writes
(aka when replication sucks)
• •
Writing doesn't scale with replication All servers needs to do the same writes
reads reads reads reads reads
Partition the data
Divide and Conquer! or Web 2.0 Buzzword Compliant! Now free with purchase of milk!!
Partition your data
• 96% read application? Skip this
Cat cluster
master slave master slave
Dog cluster
• Solution to the too many writes
problem: Don't have all data on all servers different data sets
slave slave slave slave
• Use a separate cluster for
The Write Web!
• • • • •
Replication too slow? Don't have replication slaves! Use a (fake) master-master setup and partition / shard the data! Simple redundancy! No latency from commit to data being available Don't bother with fancy 2 or 3 phase commits
(Make each "main object" (user, product, ...) always use the same master – as long as it's available)
Partition with a global master server
• • • • • •
Can't divide data up in "dogs" and "cats"? Flexible partitioning! The "global" server keeps track of which cluster has the data for user "623" Get all PKs from the global master Only auto_increment columns in the "global master" Aggressively cache the "global master" data (memcached)
webservers Where is user 623? user 623 is in cluster 3
global master
and/or use MySQL Cluster (ndb)
data clusters
select * from some_data where user_id = 623
cluster 3
cluster 2 cluster 1
Master – Master setup
• • • •
Setup two replicas of your database copying changes to each-other Keep it simple!
(all writes to one master)
Instant fail-over host – no slave changes needed Configuration is easy!
• set-variable set-variable
= auto_increment_increment=2 = auto_increment_offset=1
• •
(offset = 2 on second master) Setup both systems as a slave of the other
Online Schema Changes
The reasons we love master-master!
Do big schema changes with no downtime!
• • • • • •
Stop A to B replication Move traffic to B Do changes on A Wait for A to catchup on replication Move traffic to A Re-start A to B replication
Don't be afraid of the data-duplication monster
Summary tables!
Find queries that do things with COUNT(*) and GROUP BY and create tables with the results!
• • •
Data loading process updates both tables or hourly/daily/... updates
Variation: Duplicate data in a different "partition" Data affecting both a "user" and a "group" goes in both the "user" and the "group" partition (Flickr does
Summary databases!
• • •
Don't just create summary tables Use summary databases! Copy the data into special databases optimized for special queries
• • •
full text searches index with both cats and dogs anything spanning all clusters
Different databases for different latency requirements (RSS feeds from replicated slave DB)
Make everything repeatable
• • • •
Script failed in the middle of the nightly processing job?
(they will sooner or later, no matter what)
How do you restart it? Build your "summary" and "load" scripts so they always can be run again! (and again and again) One "authoritative" copy of a data piece – summaries and copies are (re)created from there
Asynchronous data loading
• • • • •
Updating counts? Loading logs? Don't talk directly to the database, send updates through Spread (or whatever) to a daemon loading data Don't update for each request
update counts set count=count+1 where id=37
Aggregate 1000 records or 2 minutes data and do fewer database changes
update counts set count=count+42 where id=37
Being disconnected from the DB will let the frontend keep running if the DB is down!
"Manual" replication
• • • • • • •
Save data to multiple "partitions" Application writes two places or last_updated/modified_on and deleted columns or Use triggers to add to "replication_queue" table Background program to copy data based on the queue table or the last_updated column Build summary tables or databases in this process Build star/spoke replication system
Preload, -dump and -process
Let the servers do as much as possible without touching the database directly
• • •
Data structures in memory – ultimate cache! Dump never changing data structures to JS files for the client to cache
Dump smaller read-only often accessed data sets to SQLite or BerkeleyDB and rsync to each webserver (or use NFS, but...)
Or a MySQL replica on each webserver
Stored Procedures Dangerous
• • • •
Not horizontal Bad: Work done in the database server (unless it's read-only and
Good: Work done on one of the scalable web fronts Only do stored procedures if they save the database work (network-io work > SP work)
a brief diversion ...
Running Oracle now?
• • • •
Move read operations to MySQL! Replicate from Oracle to a MySQL cluster with "manual replication" Use triggers to keep track of changed rows reads in Oracle Copy them to the MySQL master server with a replication program
Good way to "sneak" MySQL in ...
Oracle replication program writes
Optimize the database
Faster, faster, faster ....
... very briefly
• •
The whole conference here is about this ... so I'll just touch on a few ideas
Memory for MySQL = good
• • • • •
Put as much memory you can afford in the server
(Currently 2GB sticks are the best value)
InnoDB: Let MySQL use ~all memory (don't use more
than is available, of course!)
MyISAM: Leave more memory for OS page caches Can you afford to lose data on a crash? Optimize accordingly Disk setup: We'll talk about RAID later
What's your app doing?
• •
Enable query logging in your development DB! Are all those queries really necessary? Cache candidates?
(you do have a devel db, right?)
• •
Just add "log=/var/lib/mysq/sql.log" to .cnf Slow query logging:
log-slow-queries log-queries-not-using-indexes long_query_time=1 mysqldumpslow parses the slow log
5.1+ does not require a server restart and, can log directly into a CSV table...
Table Choice
• •
Short version: Use InnoDB, it's harder to make them fall over Long version: Use InnoDB except for
• • • • •
Big read-only tables (smaller, less IO) High volume streaming tables (think logging)
Locked tables / INSERT DELAYED
ARCHIVE table engine
Specialized engines for special needs More engines in the future For now: InnoDB
Multiple MySQL instances
• • • • •
Run different MySQL instances for different workloads
• •
Even when they share the same server anyway! InnoDB vs MyISAM instance
prod cluster (innodb, normalized columns)
Move to separate hardware and replication easier Optimize MySQL for the particular workload Very easy to setup with the instance manager or mysqld_multi mysql.com init.d script supports the instance manager (don't use the redhat/fedora script!)
search_load process
search cluster (myisam, fulltext columns)
Config tuning helps Query tuning works
• • • • •
Configuration tuning helps a little The big performance improvements comes from schema and query optimizations – focus on that! Design schema based on queries Think about what kind of operations will be common on the data; don't go for "perfect schema beauty" What results do you need? (now and in the future)
• • •
Use the "EXPLAIN SELECT ..." command to check the query Baron Schwartz talks about this 2pm on Tuesday! Be sure to read
http://dev.mysql.com/doc/mysql/en/mysql-indexes.html http://dev.mysql.com/doc/mysql/en/explain.html
Use smaller data
Use Integers
• •
Always use integers for join keys And when possible for sorts, group bys, comparisons
• •
Don't use bigint when int will do Don't use varchar(255) when varchar(20) will do
Store Large Binary Objects
(aka how to store images)
• • • •
Meta-data table (name, size, ...) Store images either in the file system
• • • •
meta data says "server '123', filename 'abc'" (If you want this; use mogilefs or Amazon S3 for storage!)
OR store images in other tables Split data up so each table don't get bigger than ~4GB
Include "last modified date" in meta data Include it in your URLs if possible to optimize caching
images/$timestamp/$id.jpg) (/
Reconsider Persistent DB Connections
• • • • • •
DB connection = thread = memory With partitioning all httpd processes talk to all DBs With lots of caching you might not need the main database that often MySQL connections are fast Always use persistent connections with Oracle!
• •
Commercial connection pooling products
pgsql, sybase, oracle? Need thousands of persistent connections? In Perl the new DBD::Gofer can help with pooling!
InnoDB configuration
• innodb_file_per_table • Makes optimize
Splits your innodb data into a file per table instead of one big annoying file
table `table` clear unused space
• innodb_buffer_pool_size=($MEM*0.80) • innodb_flush_log_at_trx_commit setting • innodb_log_file_size • transaction-isolation = READ-COMMITTED
My favorite MySQL feature
• • •
insert into t (somedate) values ("blah"); insert into t (someenum) values ("bad value"); Make MySQL picky about bad input!
sql_mode = 'STRICT_TRANS_TABLES';
• Make your application do this on connect
Don't overwork the DB
• • • •
Databases don't easily scale Don't make the database do a ton of work Referential integrity is good
Tons of stored procedures to validate and process data not so much
Don't be too afraid of de-normalized data – sometimes it's worth the tradeoffs (call them summary tables
and the DBAs won't notice)
Use your resources wisely
don't implode when things run warm
Work in parallel
• •
Split the work into smaller (but reasonable) pieces and run them on different boxes Send the sub-requests off as soon as possible, do something else and then retrieve the results
Job queues
• • • •
Processing time too long for the user to wait? Can only process N requests / jobs in parallel? Use queues (and external worker processes) IFRAMEs and AJAX can make this really spiffy (tell the user "the wait
time is 20 seconds")
Job queue tools
Database "queue"
• • • •
Dedicated queue table or just processed_on and grabbed_on columns Webserver submits job First available "worker" picks it up and returns the result to the queue Webserver polls for status
Queue DB
workers workers workers workers
More Job Queue tools
• beanstalkd - great protocol, fast, no persistence (yet) • gearman - for one off out-of-band jobs • starling - from twitter, memcached protocol, disk based persistence
http://xph.us/software/beanstalkd/ http://www.danga.com/gearman/ http://rubyforge.org/projects/starling/
• TheSchwartz from SixApart, used in Movable Type • Spread • MQ / Java Messaging Service(?) / ...
Log http requests
• • • • • •
Log slow http transactions to a database
time, response_time, uri, remote_ip, user_agent, request_args, user, svn_branch_revision, log_reason (a "SET" column), ...
Log to ARCHIVE tables, rotate hourly / weekly / ... Log 2% of all requests! Log all 4xx and 5xx requests Great for statistical analysis!
• •
Which requests are slower Is the site getting faster or slower?
Time::HiRes in Perl, microseconds from gettimeofday system call
• • • •
Use light processes for light tasks
Thin proxies servers or threads for "network buffers" Goes between the user and your heavier backend application Built-in load-balancing! (for Varnish, perlbal, ...) httpd with mod_proxy / mod_backhand
• •
– more on that in a bit
Varnish, squid, pound, ...
Proxy illustration
perlbal or mod_proxy low memory/resource usage
Users backends lots of memory db connections etc
Light processes
• • • • • •
Save memory and database connections This works spectacularly well. Really! Can also serve static files Avoid starting your main application as root Load balancing In particular important if your backend processes are "heavy"
Light processes
Apache 2 makes it Really Easy
ProxyPreserveHost On <VirtualHost *> ServerName combust.c2.askask.com ServerAlias *.c2.askask.com RewriteEngine on RewriteRule (.*) http://localhost:8230$1 [P] </VirtualHost>
• •
Easy to have different "backend environments" on one IP Backend setup (Apache 1.x)
Listen 127.0.0.1:8230 Port 80
perlbal configuration
CREATE POOL POOL POOL POOL POOL my_apaches my_apaches ADD 10.0.0.10:8080 my_apaches ADD 10.0.0.11:8080 my_apaches ADD 10.0.0.12 my_apaches ADD 10.0.0.13:8081 0.0.0.0:80 reverse_proxy my_apaches on on on
CREATE SERVICE balancer SET listen = SET role = SET pool = SET persist_client = SET persist_backend = SET verify_backend = ENABLE balancer
A few thoughts on development ...
All Unicode All The Time
• •
The web is international and multilingual, deal with it. All Unicode all the time!
(except when you don't need it – urls, email addresses, ...)
Perl: DBD::mysql was fixed last year! PHP 6 will have improved Unicode support. Ruby 2 will someday, too...
It will never be easier to convert than now!
Use UTC
Coordinated Universal Time
• • •
It might not seem important now, but some day ... It will never be easier to convert than now! Store all dates and times as UTC, convert to "local time" on display
Build on APIs
• • • • • •
All APIs All The Time! Use "clean APIs" Internally in your application architecture Loosely coupled APIs are easier to scale
Add versioning to APIs ("&api_version=123")
Easier to scale development Easier to scale deployment Easier to open up to partners and users!
Why APIs?
Natural place for "business logic"
• • • • • •
Controller = "Speak HTTP" Model = "Speak SQL" View = "Format HTML / ..." API = "Do Stuff"
Aggregate just the right amount of data Awesome place for optimizations that matter! The data layer knows too little
More development philosophy
• • • •
Do the Simplest Thing That Can Possibly Work ... but do it really well! Balance the complexity, err on the side of simple This is hard!
Pay your technical debt
Don't incur technical debt
• • •
"We can't change that - last we tried the site went down" "Just add a comment with 'TODO'" "Oops. Where are the backups? What do you mean 'no'?" "Who has the email with that bug?"
• •
Interest on technical debt will kill you Pay it back as soon as you can!
Coding guidelines
• • •
Keep your formatting consistent!
perl: perltidy, perl best practices, Perl::Critic
Keep your APIs and module conventions consistent Refactor APIs mercilessly (in particular while they are not public)
qmail lessons
• • • •
Lessons from 10 years of qmail Research paper from Dan Bernstein
Eliminate bugs
• •
Test coverage Keep data flow explicit
qmail lessons (2)
Eliminate code – less code = less bugs!
• • • •
Refactor common code Reuse code (Unix tools / libs, CPAN, PEAR, Ruby Gems, ...) Reuse access control
Eliminate trusted code – what needs access? Treat transformation code as completely untrusted
Joint Strike Fighter
• • • • •
~Superset of the "Motor Industry Software Reliability Association Guidelines For The Use Of The C Language In Vehicle Based Software" Really Very Detailed! No recursion!
(Ok, ignore this one :-) )
Do make guide lines – know when to break them Have code reviews - make sure every commit email gets read (and have automatic commit emails in the first place!)
High Availability
and Load Balancing and Disaster Recovery
High Availability
• • •
Automatically handle failures!
unplugged the wrong box", ...) (bad disks, failing fans, "oops,
For your app servers the load balancing system should take out "bad servers" (most do)
perlbal or Varnish can do this for http servers
Easy-ish for things that can just "run on lots of boxes"
Make that service always work!
Sometimes you need a service to always run, but on specific IP addresses
• • • • •
Load balancers (level 3 or level 7: perlbal/varnish/squid) Routers DNS servers NFS servers Anything that has failover or an alternate server – the IP needs to move (much faster than changing DNS)
Load balancing
• • • • •
Key to horizontal scaling (duh) 1) All requests goes to the load balancer 2) Load balancer picks a "real server" Hardware (lots of vendors)
Coyote Point have relatively cheaper ones
Look for older models for cheap on eBay!
Linux Virtual Server Open/FreeBSD firewall rules (pf firewall pools)
(no automatic failover, have to do that on the "real servers")
Load balancing 2
• • •
Use a "level 3" (tcp connections only) tool to send traffic to your proxies Through the proxies do "level 7" (http) load balancing perlbal has some really good features for this!
• • • • •
Event based for HTTP load balancing, web serving, and a mix of the two (see below). Practical fancy features like "multiplexing" keep-alive connections to both users and back-ends Everything can be configured or reconfigured on the fly If you configure your backends to only allow as many connections as they can handle (you should anyway!) perlbal with automatically balance the load "perfectly" Can actually give Perlbal a list of URLs to try. Perlbal will find one that's alive. Instant failover!
• • • • • • •
Modern high performance http accelerator Optimized as a "reverse cache" Whenever you would have used squid, give this a look Recently got "Vary" support Super efficient (except it really wants to "take over" a box) Written by Poul-Henning Kamp, famed FreeBSD contributor BSD licensed, work is being paid by a norwegian newspaper
• http://www.varnish-cache.org/
Fail-over tools
"move that IP"
Buy a "hardware load balancer"
• • • •
Generally Quite Expensive
(Except on eBay - used network equipment is often great)
Not appropriate (cost-wise) until you have MANY servers If the feature list fits it "Just Works" ... but when we are starting out, what do we use?
• • • • • • • •
Simple, just moves the IP(s) Can embed Perl so you can run Perl functions when IPs come and go Easy configuration format Setup "groups of IPs" Supports Linux, FreeBSD and Solaris Spread toolkit for communication Easy to troubleshoot (after you get Spread working...) http://www.backhand.org/wackamole/
• • • • • •
Monitors and moves services (an IP address is "just a service") v1 has simple but goofy configuration format v2 supports all sorts of groupings, larger clusters (up to 16 servers) Uses /etc/init.d type scripts for running services Maybe more complicated than you want your HA tools http://www.linux-ha.org/
Carp + pfsync
Patent-free version of Ciscos "VRRP" (Virtual Router Redundancy Protocol)
• FreeBSD and OpenBSD only • Carp (moves IPs) and pfsync (synchronizes firewall state) • (awesome for routers and NAT boxes) • Doesn't do any service checks, just moves IPs around
mysql master master replication manager
• • • • • •
mysql-master-master tool can do automatic failover! No shared disk Define potential "readers" and "writers" List of "application access" IPs Reconfigures replication Moves IPs
http://code.google.com/p/mysql-master-master/ http://groups.google.com/group/mmm-devel/
Suggested Configuration
• •
Open/FreeBSD routers with Carp+pfsync for firewalls A set of boxes with perlbal + wackamole on static "always up" HTTP enabled IPs
Trick on Linux: Allow the perlbal processes to bind to all IPs (no port number tricks or service reconfiguration or restarts!)
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
or or
sysctl -w net.ipv4.ip_nonlocal_bind=1 echo net.ipv4.ip_nonlocal_bind = 1 >> /etc/sysctl.conf
• • •
Dumb regular http servers "behind" the perlbal ones wackamole for other services like DNS mmm for mysql fail-over
Redundancy fallacy!
• •
Don't confuse load-balancing with redundancy What happens when one of these two fail?
Load balanced servers
load / capacity
Load (55%)
Load (60%)
Oops – no redundancy!
• • •
Always have "n+1" capacity Consider have a "passive spare" (active/passive with two servers) Careful load monitoring!
More than 100% load on 1 server!
Load (50%)
• • •
Munin http://munin.projects.linpro.no/ MySQL Network (ganglia, cacti, ...)
Load (60%)
High availability Shared storage
• • • • •
NFS servers (for diskless servers, ...) Failover for database servers Traditionally either via fiber or SCSI connected to both servers Or NetApp filer boxes All expensive and smells like "the one big server"
Cheap high availability storage with DRBD
• • • • •
Synchronizes a block device between two servers! "Network RAID1" Typically used in Active/Primary-Standby/Secondary setup If the active server goes down the secondary server will switch to primary, run fsck, mount the device and start the service (MySQL / NFS server / ...) v0.8 can do writes on both servers at once – "shared disk semantics" (you need a filesystem on top that supports that, OCFS, GFS, ... –
probably not worth it, but neat)
Disaster Recovery
Separate from "fail-over" (no disaster if we failed-over...)
• • • •
"The rescue truck fell in the water" "All the 'redundant' network cables melted" "The datacenter got flooded" "The grumpy sysadmin sabotaged everything before he left"
Disaster Recovery Planning
• • • • •
You won't be back up in 2 hours, but plan so you quickly will have an idea how long it will be Have a status update site / weblog Plans for getting hardware replacements Plans for getting running temporarily on rented "dedicated servers" (ev1servers, rackspace, ...) And ....
Backup your databse!
• • • •
Binary logs!
Keep track of "changes since the last snapshot"
Use replication to Another Site
(doesn't help on "for $table = @tables { truncate $table }")
On small databases use mysqldump Zmanda MySQL Backup
(or whatever similar tool your database comes with)
packages the different tools and options
Backup Big Databases
Use mylvmbackup to snapshot and archive
• • • • •
Requires data on an LVM device (just do it) InnoDB: Automatic recovery!
(ooh, magic)
MyISAM: Read Lock your database for a few seconds before making the snapshot
(on MySQL do a "FLUSH TABLES" first (which might be slow) and then a "FLUSH TABLES WITH READ LOCK" right after)
Sync the LVM snapshot elsewhere And then remove the snapshot!
Bonus Optimization: Run the backup from a replication slave!
Backup on replication slave
• •
Or just run the backup from a replication slave ... Keep an extra replica of your master
• •
shutdown mysqld and archive the data Small-ish databases:
mysqldump --single-transaction
All Automation All The Time or How to manage 200 servers in your spare-time
System Management
Keep software deployments easy
• • •
Make upgrading the software a simple process Script database schema changes Keep configuration minimal
• • • •
Servername ("www.example.com") Database names ("userdb = host=db1;db=users";..." If there's a reasonable default, put the default in the code (for
example )
"deployment_mode = devel / test / prod" lets you put reasonable defaults in code
Easy software deployment 2
• • • • • •
How do you distribute your code to all the app servers? Use your source code repository (Subversion etc)! (tell your
script to svn up to http://svn/branches/prod revision 123 and restart)
.tar.gz to be unpacked on each server .rpm or .deb package NFS mount and symlinks No matter what: Make your test environment use the same mechanism as production and: Have it scripted!
have everything scripted!
Configuration management
Rule Number One
• • • •
Configuration in SVN (or similar)
"infrastructure/" repository SVN rather than rcs to automatically have a backup in the Subversion server – which you are carefully backing up anyway Keep notes! Accessible when the wiki is down; easy to grep Don't worry about perfect layout; just keep it updated
Configuration management
Rule Two
• • • •
Repeatable configuration! Can you reinstall any server Right Now? Use tools to keep system configuration in sync Upcoming configuration management (and more) tools!
• •
csync2 (librsync and sqlite based sync tool) puppet (central server, rule system, ruby!)
• Automating sysadmin tasks! • 1) Client provides "facter" to server
2) Server makes configuration 3) Client implements configuration
service { "sshd": enable => true, ensure => running } package { "vim-enhanced": ensure => installed } package { "emacs": ensure => installed }
• •
node db-server inherits standard { include mysql_server include solfo_hw }
puppet example
node db2, db3, db4 inherits db-server { } node trillian inherits db-server { include ypbot_devel_dependencies } # ----------------------------class mysql_client { package { "MySQL-client-standard": ensure => installed } package { "MySQL-shared-compat": ensure => installed } } class mysql_server { file { "/mysql": ensure => directory, } package { "MySQL-server-standard": ensure => installed } } include mysql_client
puppet mount example
class nfs_client_pkg {
Ensure an NFS mount exists, except on the NFS servers
file { "/pkg": ensure => directory, } $mount = $hostname ? { "nfs-a" => absent, "nfs-b" => absent, default => mounted } mount { "/pkg": atboot => true, device => 'nfs.la.sol:/pkg', ensure => $mount, fstype => 'nfs4', options => 'ro,intr,noatime', require => File["/pkg"], } }
More puppet features
In addition to services, packages and mounts...
• • • • •
Manage users Manage crontabs Copy configuration files (with templates) … and much more
Recipes, reference documentation and more at
• •
Backup everything you can
• • • • • •
Check/test the backups routinely
Super easy deployment: rsnapshot Uses rsync and hardlinks to efficiently store many backup generations Server initiated – just needs ssh and rsync on client Simple restore – files
Other tools Amanda (Zmanda) Bacula
Backup is cheap!
• • •
Extra disk in a box somewhere? That can do! Disks are cheap – get more! Disk backup server in your office: Enclosure + PSU: $275 CPU + Board + RAM: $400 3ware raid (optional): $575 6x1TB disks: $1700 (~4TB in raid 6) = $3000 for 4TB backup space, easily expandable
(or less than $5000 for 9TB space with raid 6 and hot standby)
Ability to get back your data = Priceless!
somewhat tangentially ...
RAID Levels
RAID-I (1989) consisted of a Sun 4/280 workstation with 128 MB of DRAM, four dualstring SCSI controllers, 28 5.25-inch SCSI disks and specialized disk striping software.
Basic RAID levels
• • • • •
RAID 0 Stripe all disks (capacity = N*S Fail: Any disk RAID 1 Mirror all disks (capacity = S) Fail: All disks RAID 10 Combine RAID 1 and 0 (capacity = N*S / 2) RAID 5 RAID 0 with parity (capacity = N*S - S) Fail: 2 disks RAID 6 Two parity disks (capacity = N*S - S*2) Fail: 3 disks!
• • •
Mirror all disks to all disks Simple - easiest to recover! Use for system disks and small backup devices
• • • • • •
Use for redundant database mirrors or scratch data that you can quickly rebuild Absolutely never for anything you care about Failure = system failure Great performance; no safety Capacity = 100% Disk IO = every IO available is "useful"
• • • • •
Stripe of mirrored devices IO performance and capacity of half your disks - not bad! Relatively good redundancy, lose one disk from each of the "sub-mirrors" Quick rebuild: Just rebuild one mirror More disks = more failures! If you have more than X disks, keep a hot spare.
• • • • •
Terrible database performance A partial block write = read all disks! When degraded a RAID 5 is a RAID 0 in redundancy! Rebuilding a RAID 5 is a great way to find more latent errors Don't use RAID 5 – just not worth it
• • •
Like RAID 5 but doesn't fail as easily Can survive two disks failing Don't make your arrays too big
• •
12 disks = 12x failure rate of one disk! Always keep a hot-spare if you can
Hardware or software RAID?
• •
Hardware RAID: Worth it for the Battery Backup Unit!
• • • •
Battery allows the controller to – safely – fake "Sure mister, it's safely on disk" responses
No Battery? Use Software RAID Low or no CPU use Easier and faster to recover from failures!
• •
Write-intent bitmap
More flexible layout options RAID 1 partition for system + RAID 10 for data on each disk
• • • •
Monitoring "is the website up" is easy Monitoring dozens or hundreds of sub-systems is hard Monitor everything! Disk usage, system daemons, applications daemons, databases, data states, ...
nagios configuration tricks
• •
nagios configuration is famously painful Somewhat undeserved!
examples of simple configuration - templates - groups
nagios best practices
• • • •
All alerts must be "important" – if some alerts are ignored, all other alerts easily are, too. Don't get 1000 alerts if a DB server is down Don't get paged if 1 of 50 webservers crashed Why do you as a non-sysadmin care?
• •
Use nagios to help the sysadmins fix the application Get information to improve reliability
Resource management
• •
If possible, only run one service per server (makes monitoring/
managing your capacity much easier)
Balance how you use the hardware
• • • •
Use memory to save CPU or IO Balance your resource use (CPU vs RAM vs IO) Extra memory on the app server? Run memcached! Extra CPU + memory? Run an application server in a Xen box!
Don't swap memory to disk. Ever.
Netboot your application servers!
• • • • •
Definitely netboot the installation (you'll never buy another
server with a tedious CD/DVD drive)
RHEL / Fedora: Kickstart + puppet = from box to all running in ~10 minutes
Netboot application servers FreeBSD has awesome support for this Debian is supposed to Fedora Core 7 8 ?? looks like it will (RHEL5uX too?)
No shooting in foot!
Ooops? Did that leak memory again? Development server went kaboom?
• Edit /etc/security/limits.conf soft rss • @users
@users @users hard rss hard as
250000 250000 500000
Use to set higher open files limits for mysqld etc, too!
noatime mounts
• • • •
Mount ~all your filesystems "noatime" By default the filesystem will do a write every time it accesses/reads a file! That's clearly insane Stop the madness, mount noatime
/home /home ext3 ext3 defaults noatime 1 2 1 2
/dev/vg0/lvhome /dev/vg0/lvhome
graph everything!
• •
The Multi Router Traffic Grapher
round-robin-database tool
• •
Fixed size database handling time series data Lots of tools built on rrdtool
cluster/grid monitoring system
Historical perspective
basic bandwidth graph
Steady growth Try CDN Enable compression for all browsers
• • • •
"Hugin and Munin are the ravens of the Norse god king Odin. They flew all over Midgard for him, seeing and remembering, and later telling him." Munin is also AWESOME! Shows trends for system statistics Easy to extend
mysql query stats
Query cache useful?
• • •
Is the MySQL query cache useful for your application? Make a graph! In this particular installation it answers half of the selects
squid cache hitratio?
• • • •
Red: Cache Miss Green: Cache Hit Increased cache size to get better hit ratio Huh? When? Don't confuse graphs with "hard data" Keep the real numbers, too!
munin: capacity planning, cpu
• •
xen system 6 cpus plenty to spare
Blocking on disk I/O?
• •
Pink: iowait This box needs more memory or faster disks!
More IO Wait fun
• •
8 CPU box - harder to see the details High IO Wait
More IO Wait fun
Upgraded memory, iowait dropped!
IO Statistics
• •
per disk IO statistics more memory, less disk IO
more memory stats
fix app config
plenty memory free
fix perlbal leak
room for memcached?
took a week to use new memory for caching plenty memory to run memcached here!
munin: spot a problem?
• •
1 CPU 100% busy on "system"? Started a few days ago
munin: spot a problem?
• •
Has it happened before? Yup - occasionally!
munin: spot a problem?
IPMI driver went kaboom!
Make your own Munin plugin
Any executable with the right output
# ./load config graph_title Load average graph_args --base 1000 -l 0 graph_vlabel load ... load.label load load.info Average load for the five minutes. ... # ./load fetch load.value 1.67
Munin as a nagios agent
• • •
Use a Nagios plugin to talk to munin! Munin is already setup to monitor important metrics Nagios plugin talks to munin as if the collector agent
define service { use local-service hostgroup_name xen-servers,db-servers,app-ser service_description df check_command check_munin!df!88!94 }
A little on hardware
• • • • • • • •
Hardware is a commodity! Configuring it isn't (yet – Google AppEngine!) Managed services - cthought.com, RackSpace, SoftLayer ... Managing hardware != Managing systems Rent A Server
(crummy support, easy on hardware replacements, easy on cashflow)
Amazon EC2 (just announced persistent storage!) Use standard configurations and automatic deployment Now you can buy or rent servers from anywhere!
Use a CDN
• • • • •
If you serve more than a few TB static files a month... Consider a Content Delivery Network Fast for users, easier on your network Pass-through proxy cache - easy deployment Akamai, LimeLight, PantherExpress, CacheFly, ...
(only Akamai supports compressed files (??))
Client Performance
"Best Practices for Speeding Up Your Web Site"
Recommended Reading
"High Performance Web Sites" book by Steve Souders
• http://developer.yahoo.com /performance/
• • •
Use YSlow
Firefox extension made by Yahoo! Quickly checks your site for the Yahoo Performance Guidelines I'll quickly go over a few server / infrastructure related rules ...
• http://developer.yahoo.com/yslow/
Minimize HTTP Requests
• • •
Generate and download the main html in 0.3 seconds Making connections and downloading 38 small dependencies (CSS, JS, PNG, …) – more than 0.3s! Combine small JS and CSS files into fewer larger files
• •
Make it part of your release process! In development use many small files, in production group them
CSS sprites to minimize image requests
Add an "Expires" header
• • • •
Avoid unnecessary "yup, that hasn't changed" requests Tell the browser to cache objects HTTP headers
Expires: Mon, Jan 28 2019 23:45:00 GMT Cache-Control: max-age=315360000
Must change the URL when the file changes!
Ultimate Cache Control
• • • • • •
Have all your static resources be truly static Change the URL when the resource changes Version number – from Subversion, git, …
/js/foo.v1.js /js/foo.v2.js ...
Modified timestamp – good for development
(partial) MD5 of file contents – safe for cache poisoning
Build a "file to version" mapping in your build process and load in the application
Serve "versioned" files
• • •
Crazy easy with Apache rewrite rules "/js/foo.js" is served normally "/js/foo.vX.js" is served with extra cache headers
RewriteEngine on # remove version number, set environment variable RewriteRule ^/(.*\.)v[0-9a-f.]+\.(css|js|gif|png|jpg|ico)$ \ /$1$2 [E=VERSIONED_FILE:1] # Set headers when "VERSIONED_FILE" environment is set Header add "Expires" "Fri, Nov 10 2017 23:45:00 GMT" \ env=VERSIONED_FILE Header add "Cache-Control" "max-age=315360001" \ env=VERSIONED_FILE
Minimize CSS, JS and PNG
Minimize JS and CSS files (remove whitespace, shorten JS, …)
• •
Add to your "version map" if you have a "-min" version of the file to be used in production Losslessly recompress PNG files with OptiPNG
function EventsFunctions() { this.get_data = function(loc_id) { if (this.TIMEOUT) { window.clearTimeout(this.TIMEOUT); this.TIMEOUT = null; } var parameters = 'auth_token=' + escape(global_auth_token) + ';total=5;location='+loc_id; var request = YAHOO.util.Connect.asyncRequest('POST', '/api/events/location_events', { success:function(o) { var response = eval( '(' + o.responseText + ')' ); if (response.system_error) { // alert(response.system_error); } else if (response.length) { var eventshtml=''; for (var i=0; i<response.length; i++) { eventshtml+='<br /><a href="http://example.com/event/'+response[i].id+'/">'+ response[i].name+'</a> - '+response[i].start_date; if (response[i].start_time) eventshtml+=' '+response[i].start_time; if (response[i].description) eventshtml+='<br />'+response[i].description; eventshtml+='<br /><br />'; } var le = document.createElement("DIV"); le.id='location_events'; le.innerHTML=eventshtml; document.body.appendChild(le); tab_lookups['events_tab'] = new YAHOO.widget.Tab({ label: 'Events', contentEl: document.getElementById('location_events') }); profileTabs.addTab(tab_lookups['events_tab']); } try{ pageTracker._trackPageview('/api/events/location_events') } catch(err) {} }, failure:function(o) { // error contacting server }
Pre-minimized JS
~1600 to ~1100 bytes ~30% saved!
Minimized JS
function EventsFunctions(){this.get_data=function(loc_id){if(this.TIMEOUT) {window.clearTimeout(this.TIMEOUT); this.TIMEOUT=null;}var parameters="auth_token="+escape(global_auth_token) +";total=5;location="+loc_id; var request=YAHOO.util.Connect.asyncRequest("POST","/api/events/location_events", {success:function(o){var response=eval("("+o.responseText+")"); if(response.system_error){}else{if(response.length){var eventshtml="";for(var i=0;i<response.length;i++){eventshtml+='<br /><a href="http://example.com/ event/'+response[i].id+'/">'+response[i].name+"</a> - "+response[i].start_date; if(response[i].start_time){eventshtml+=" "+response[i].start_time;}if(response[i].description){eventshtml+="<br / >"+response[i].description; }eventshtml+="<br /><br />";}var le=document.createElement("DIV");le.id="location_events";le.innerHTML=eventshtml; document.body.appendChild(le);tab_lookups.events_tab=new YAHOO.widget.Tab({label:"Events",contentEl:document.getElementById("location_events")}); profileTabs.addTab(tab_lookups.events_tab);}}try{pageTracker._trackPageview("/api/events/ location_events");
Gzip components
• • • •
Don't make the users download several times more data than necessary. Browser:
Accept-Encoding: gzip, deflate
Content-Encoding: gzip
Dynamic content (Apache 2.x)
LoadModule mod_deflate … AddOutputFilterByType DEFLATE text/html text/plain text/javascript text/xml
Gzip static objects
Pre-compress .js and .css files in the build process foo.js > foo.js.gzip
AddEncoding gzip .gzip # If the user accepts gzip data RewriteCond %{HTTP:Accept-Encoding} gzip # … and we have a .gzip version of the file RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME}.gzip -f # then serve that instead of the original file RewriteRule ^(.*)$ $1.gzip [L]
Think Horizontal!
(and go
build som
ething n
• • •
"Building Scalable Web Sites" by Cal Henderson of Flickr fame
Only $26 on Amazon!
from your local bookstore too)
(But it's worth the $40
"Scalable Internet Architectures" by Theo Schlossnagle Teaching concepts with lots of examples "High Performance Web Sites" by Steve Souders Front end performance
• • • • • • • • • • • • • •
Direct and indirect help from ...
Cal Henderson, Flickr Yahoo! Brad Fitzpatrick, LiveJournal SixApart Google Graham Barr Tim Bunce Perrin Harkins David Wheeler Tom Metro Kevin Scaldeferri, Overture Yahoo! Vani Raja Hansen Jay Pipes Joshua Schachter Ticketmaster Shopzilla .. and many more
– The End –
Questions? Thank you!
More questions? Comments? Need consulting? firstname.lastname@example.com


text:
Mon-Sat: 10:00 AM - 9:30 PM
Sun: 11:00 AM - 7:00 PM
Restaurant, Theater and Department Store Hours May Vary
Just minutes from downtown Washington, D.C., Tysons Corner Center is the Nation's Capital choice for selection, style and fashion. World-class shopping, dining and entertainment converge to offer the visitor a monumental experience. Tysons Corner Center is located about seven miles west of Washington, D.C. just off the Capital Beltway (I-495) near the intersection of Routes 7 and 123. You can also jump on the METRO Silver Line and get off at our designated stop.
As one of the top 10 shopping centers in the country, Tysons Corner Center offers an unbeatable mix of more than 300 department and specialty stores. Tysons' "who's who" of retailers includes Nordstrom, Bloomingdale's, Macy's, Lord & Taylor, American Girl, L.L. Bean, Uniqlo, Louis Vuitton, Gucci, Michael Kors, Lacoste, Spanx, Apple, The Disney Store and more. Tysons Corner Center also offers a 16-screen stadium seating and IMAX 3D AMC Theatre, fine and casual dining restaurants, a food court and a National Geographic sponsored children's play area. The outdoor Plaza at Tysons Corner hosts events throughout the year and provides a number of eateries with cafe seating. The 310 room luxury Hyatt Regency Hotel at Tysons Corner Center is the perfect respite from a busy day of shopping.
Show your AAA/CAA card and your out-of-town license at the Concierge Desk on Level 1 in Macy's Court to receive your special discount offers.
Tysons Corner Center is home to national and international brands including:
Dining options include:

subset: commoncrawl


text:
Added May 6, 2009
by yoshimi
, Japan
I made two parasols in my favorite fabrics. The pink one is a birthday gift to my sister. I bought the DIY parasol kits which contained the frame and the instruction. I just sewed my fabrics to make canopies. I really love what I've got after struggling with my first DIY parasol experience.
Material Notes
DIY parasol kits, cotton from marimekko
blogged here
15 Comments Sign in to add a post
Sep 14, 2010, 02.52 PMby mkim630
wow, i absolutely love these! i totally need to find some parasol DIY kits!
Sep 3, 2010, 08.24 AMby Pauline Pepsi
Wow beautiful!
Sep 24, 2009, 09.31 PMby ichigogirl
I LOVE the black and white one… makes me want to visit Tokyu Hands. Maybe you didn't get the kit from there but your gorgeous parasols make me remember all the wonderful things there… thanks for sweet nostalgia and happy memories! Hope your sister was happy with the lovely gift!
Aug 31, 2009, 09.55 PMby sewyoutogethergoodasnew
thats so cool! no clue how you did it but they look amazing!!
Jun 26, 2009, 06.10 PMby dressdummy
That's so awesome! Umbrellas and parasols are so interesting anyway, in all times and all places. Yours is beautiful. Let me ask you this: Now that you've made one from a kit, do you think you would be able to rehabilitate old umbrella frames using new fabric?
Jun 24, 2009, 01.00 PMby susanne2011
Wow, I really love the fabric with the black leaves!
May 8, 2009, 09.37 AMby pattyrg
Wow great job! I dont think I would have the patience to do one!
May 8, 2009, 03.20 AMby carottesauvage
That's amazing Yoshimi! Thanks for the inspiration!
May 7, 2009, 01.06 AMby lybu
wow. They are just SUPER. Great idea. I want to try it out to :)
May 6, 2009, 07.56 AMby hajnikaOh,these are greats,and what a beautiful fabrics!!
May 6, 2009, 07.31 AMby soozy
Oh! they are adorable! My to do/wish list is allready way to long, but maybe I'll look up a DIY kit too. You're gonna look great together with your parasols.
May 6, 2009, 06.43 AMby meseide
I love these and wish I were your sister. :) Where did you buy the DIY kits?
May 6, 2009, 05.18 AMby ralarkins
Oh, Can I have one too???? You did a great job on those, they look lovely. Your sister is a lucky girl. Now I am going to have to look up DIY parasol kits.
May 6, 2009, 04.59 AMby miss-g
Wow. I didn't know it was possible to make a parasol from a kit. Yours look amazing.
May 6, 2009, 04.56 AMby luzie
Wow, they are beautiful!!! Where did you buy the parasol kits? Did you put something on to make them waterproof or do you just use them for the sun? I like the fabrics too, you got a lucky sister! :-)
Add to Favorites 45 people added it


text:
Doug McDermott wasn't scoring, Evansville was coming back, and Creighton's fans were getting nervous.
As if on cue, the returning All-American found his rhythm again, scoring 14 of his 29 points in a 6½-minute stretch of the second half to help No. 16 Creighton pull away for an 87-70 victory on Saturday night.
The Purple Aces had cut Creighton's lead to 10 points early in the second half when Grant Gibbs missed a jumper that McDermott put back to end a drought that lasted 10½ minutes. By the time McDermott's flurry ended, the Bluejays led by 20 and were on their way to a sixth straight win.
"They did a really good job of defending me, were really physical with me and crowded the lane like they usually do," McDermott said. "I wasn't worried about scoring because if they put all that attention on me, it just opens the floor up for my teammates. Once I got that putback, it got me to play a little harder on both ends."
Gregory Echenique had 13 points and 13 rebounds, Jahenns Manigat added 13 points, and Avery Dingman had 11 for the Bluejays (12-1), who won their Missouri Valley Conference opener.
McDermott also had 10 rebounds for his second double-double of the season.
Colt Ryan, who scored an arena-record 43 points in his last visit, matched his season high with 25 for the Purple Aces (7-6, 0-1).
"We just couldn't let Colt have a night like last year," Creighton coach Greg McDermott said.
Troy Taylor added 10 for Evansville, which has lost 14 straight in Omaha since 1999. Taylor also had 10 rebounds and 10 assists for his first triple-double.
The Bluejays planned on a tough night. They lost 65-57 in Evansville last season, then had to rally from seven points down in the last two minutes before winning 93-92 in overtime in Omaha.
Creighton beat Evansville 99-71 in the MVC tournament semifinals.
"We have a pretty old group. We have a lot of the same guys back, so I think that helped a little bit," Echenique said. "I think trying to use that past experience, we knew they were going to come ready. They showed us at their place when we played them. We didn't want it to happen again."
Still, the Bluejays squandered most of a 17-point lead and led by just seven early in the second half.
"We didn't have much energy," Doug McDermott said. "We weren't working as hard as we were to start the game. We kind of got complacent and weren't guarding them with as much urgency as we should have."
Greg McDermott said he figured it was a matter of time before his son started scoring again after his mini funk.
"He has a way of letting the game come to him and to try not to force the issue," Coach McDermott said. "He understands that if he posts up with some urgency and some effort, it's tough for one guy to guard him."
Evansville coach Marty Simmons knows that all too well.
"We tried different guys on him, but he's one of the best players in the country and there's a reason why," Simmons said. "We wanted to limit his touches, didn't want to foul him, and keep him off the offensive glass. You look at his stats, and we didn't execute. Give him credit. He does it every night."
Creighton didn't show much rust in its first game in 10 days. For the third straight game, the Bluejays never trailed. They held a 43-27 rebounding advantage against the undersized Aces.
Evansville, which hadn't played in a week, started 5 of 20 from the field and fell behind by double digits less than 6 minutes in.
The Bluejays broke the game open with an 11-0 run that Manigat finished with a dazzling drive to the hoop.
Evansville had trouble getting Ryan touches early as Manigat, Gibbs and Dingman took turns defending him. Ryan, who scored 25 in a loss at Butler on Dec. 22, missed four of his first five shots but warmed up the last 5 minutes of the half to help the Aces get the deficit down to 11 at the break.
Evansville got it to 45-38 on Ryan's jumper in the lane and Taylor's layup off a steal early in the second half.
Chatman's 3-pointer restored the double-digit lead, but just briefly. Ryan hit a jumper from the wing, and there was a sense in the sellout crowd of 18,458 — Creighton's second-largest ever — that the Aces weren't going to go away.
But there wasn't much they could do once McDermott found his rhythm, and the Bluejays went on to their sixth straight win.
The victory bodes well for Valley favorite Creighton. Since the 1993-94 season, only once has a team won or shared the conference's regular-season championship without winning its league opener.


text:
Lawn Ornaments
Randy, Deb, and Brian Grinde of DeForest, Wisconsin have turned a hobby into a business. Several years ago, while searching for lawn ornaments for their yard, they were disappointed in the selection and quality of items they found. So they decided to make their own. Soon, friends, neighbors, and passers-by asked if they could order the items they made. Eventually their orders grew and they decided to start their own business, Accent Corner.
The Grinde's support for the University of Wisconsin led them to focus initially on Badger sports graphics. Shapes including Bucky Badger, a football helmet, and the motion "W" can be hung from a hook, attached to an aluminum stake, or attached to a building, fence, or other flat surface inside or out. The Grindes have worked hard to research equipment and materials and to improve on their products. The quality, artistic detail, and durability of their yard decorations are sure to please everyone who displays them.


text:
Martin D-18
The Martin D-18 is a masterpiece of steel-string acoustic guitar design, a modern classic descended from the earliest dreadnought models which Martin first developed almost a century ago. Martin's guitar-making history dates back even further, to 1833, when young German immigrant C.F. Martin founded the company in New York City. Five years later, he moved operations to Nazareth, Pennsylvania, where the family-owned business is still located today, run by the founder's great-great-great-grandson, C.F. Chris Martin IV. The earliest Martin guitars were mostly smaller-bodied, parlor-style gut-string instruments common to the period, intended to be played solo or in small ensembles. By the late 1800s, with the advent of recorded music and the increasing popularity of larger bands, several manufacturers began offering steel-string acoustic guitars, which provided more volume and projection than gut-string instruments, but required more substantial internal bracing. In 1916, the Oliver Ditson music publishing company partnered with Martin to develop a large-bodied steel-string guitar, which, because it was deemed to be as loud as a cannon, they called the Dreadnought, named for the WW-I-era class of battleships, at the time considered to be the most formidable weapons on earth. These early dreadnoughts featured an elongated body with a spruce top and mahogany back and sides that met the neck at the 12th fret, and sported a slotted headstock. The Ditson-branded dreadnoughts were not big sellers, however, and after that company went out of business, Martin began selling their own D-series dreadnoughts. The first few years of production maintained the initial 12-fret-clear-of-the-body design, but by 1934, the familiar 14-fret-clear dreadnought design was released, and it is now the most popular steel-string acoustic guitar design in the world, providing excellent tonal balance, projection and playability.
Today, the Martin D-18 is a mainstay of the company's product line, along with dreadnought siblings like the D-28, D-35 and top-of-the-line D-45. While the D-18 sports less ornamentation than the others, (the higher the model number, the more ornate the inlays, binding and other appointments), it is by no means a lesser guitar. It features a back and sides of mahogany, as opposed to the rosewood found on most versions of the other models, which imparts its own, unique tonal characteristics, resulting in a somewhat brighter sound, well-suited for fingerpicking and playing leads. Martin offers several variations, including the standard D-18 with a solid Sitka spruce top; the Retro D-18, based on the iconic pre-war models and loaded with Fishman electronics for amplified picking; the D-18 1937, which features period-correct materials and construction methods, right down to the hide glue; and the D-18GE Golden Era 1934, based on another pre-war classic and featuring a slightly stiffer Adirondack solid spruce top. Whatever your playing style, it can only be enhanced when you play the best guitars available. Martin originated the dreadnought design and perfected it, and the D-18 is a prime example of their years of fine craftsmanship and commitment to quality. Order your Martin D-18 today and own a masterpiece.
Our product catalog varies by country due to manufacturer restrictions. If you change the Ship-To country, some or all of the items in your cart may not ship to the new destination.


text:
Release Date: October 18, 2012
BUFFALO, N.Y. -- The University at Buffalo will open to the public some of its most unique collections, including what's believed to be the nation's only museum dedicated exclusively to the human brain.
Collections of items such as medical books from the Middle Ages, Colonial-era dentures, tonics, potions and, of course, human brains will be on view from 4-7 p.m. on Oct. 24 in the following museums and collections on UB's South Campus:
-- The Museum of Neuroanatomy (AKA Brain Museum) -- Located in the Biomedical Education Building, the collection features stunning displays of the most complex and mysterious organ found in humans: brains. Built in 1994, the museum is an educational tool for everyone, from kindergartners to neurosurgery students.
-- The School of Pharmacy and Pharmaceutical Sciences Apothecary and Historical Collections -- Located in Kapoor Hall, UB's newly revamped $62 million gem, this collection includes artifacts that date back to the 18th century. Highlights include cigarettes once used to cure asthma and hair-growing tonic from Lockport's famous Sutherland Sisters.
-- The Museum of Radiology and Medical Physics -- Located in the Abbott Hall (Health Sciences Library), this collection features several hundred items including World War II era portable X-ray unit and original gas X-ray tubes.
-- The Robert L. Brown History of the Health Sciences Collection -- Located in Abbott Hall, this collection includes rare books dating back to 1493 and more than 200 medical instruments from Roman times to the late 20th century.
-- The George W. Ferry Dental Museum (note: this collection will be open from 4-6 p.m.) -- Located in Squire Hall, this collection includes a replica of a dentist's office, circa 1900, a set of ivory-carved dentures from the 17th century and other dental artifacts.
In addition, UB will host a panel discussion from 6:45-7:45 p.m. in the Roswell Park Room in the Health Sciences Library about the importance of the history of medicine for medical humanities, changing trends in psychiatry and other topics. The discussion will feature faculty members from the UB Department of History: James Bono, PhD, associate professor; and Michael Rembis, PhD, and David Herzberg, PhD, assistant professors.
Visitors will not need a parking permit, and are advised to park in the Diefendorf lot, located off Sherman Road. Note: Sherman Road is off Bailey Avenue, near the Veterans Administration hospital. For maps of UB's South Campus and further instructions about parking, visit: http://www.buffalo.edu/buildings.
Visitors wishing to attend the discussion are encouraged to RSVP by contacting Linda Lohr (829-5737, email@example.com) or Keith Mages (829-5737, email@example.com) in the Robert L. Brown History of Health Sciences Collection.
Cory Nealon
Director of News Content
Engineering, Computer Science
Tel: 716-645-4614
Twitter: @UBengineering


text:
Swishmax template jobs
I need a mail merge template in InDesign (CS4) gathering information from a filter on an excel spreadsheet. The document will be A4 and contain labels. The colour (CMYK) for each label will be defined in the spreadsheet. There are fourpieces of information that need to be includws on the label from the spreadsheet, as shown in the image. The text in
I am one of the owners of a kid's retail store ([login to view URL]) and I need a well done power point template for institutional presentation. The idea is to develop 3-4 institutional powerpoint slides we can use in business presentations. The company already has 4 mascots, and I do have the image in high resolution, if necessary
_we need to work by remote^desktop (better if in midnight indian time) _we furnish template realized with indesign _we fusnish fotos and description of company and 2 products _you place in the right position and customize with our brand and all necessary modifies _2 hours of work more or less x 5$hours = 10$ (we can pass various work as this in one
Design a Microsoft Word template for workplace procedures which includes a range of text, coloured blocks, flowcharts and tables. Demonstration documents (CPG and DTP) are provided from another similar company to visualise the kind of data which needs to be included in the template. Our letterhead/logo is included. The theme is mostly blue and
If you open the excel document attached, you'll notice there are 5 columns: queries, clicks, impressions, CTR (click through rate) and position for a fictional bike company called "example". This is example data downloaded from Google Search Console about a fictional bike company called "Example". Google Search Console allows us to download data about our website to see ...


text: Many people who are new to an exercise program or who are increasing the intensity of their workouts feel a variety of aches and pains. What's normal? When do you need to seek treatment or tone down your exercise program? As an exercise physiologist at Good Shepherd's Optimal Fitness gym, I can provide a few guidelines. But whenever you are in doubt about pain or an injury, see your physician.


text:
Tuesday , May 15, 2018 - 7:17 PM
SYRACUSE — The weather is heating up and, apparently, so is Ashtyn Bauerle's pitching.
What better way to christen the first round of the 6A state softball playoffs by throwing your first perfect game?
The Syracuse senior pitcher retired every Westlake batter that came to the plate as Bauerle tossed a five-inning perfect game in the Titans' 10-0, run-rule win over the Thunder on Tuesday.
Apparently, she was so focused that she didn't know about the perfect game until afterward.
"I've thrown no-hitters, I've thrown a couple of those — but I've never thrown a perfect game, so that was pretty cool," she said.
She struck out nine of the 15 batters she faced Tuesday, including all three Thunder batters in the first and third innings.
The only ball to leave the infield was a fly ball to left field.
The only inning where Bauerle didn't record a strikeout was the fourth, when first baseman Kylie Derrick made a nice defensive play on a short-hopped throw to preserve the perfect game.
"The big thing with her is we gotta get ahead (in the count) so we can start nibbling on the corners a little bit," head coach Kelly Anderson said. "As it's gotten warmer, she's gotten better."
"The last few games have been her best games coming down the stretch."
Syracuse gave Bauerle a big lead to work with early, plating six runs in the first inning.
Kylie Derrick 's two-run single started the rout and the other four runs that frame came across from wild pitches and passed balls. Westlake didn't record an out until it was 4-0.
Aubri Huffaker and Fallyn Marshall each drove in a run after a fielding error scored the first Syracuse run of the second inning.
That made it 9-0. Meanwhile, Bauerle continued to sit batters down left and right.
"One of our mottos this year is 'It's just a game,' because early in the year we kind of got really tense in those big games," Bauerle said. "I think that's helped us a lot."
Kemri Curtis guided an RBI single through the left side for the walkoff win in the fifth inning while Bauerle was standing in the on-deck circle.
Syracuse High and Westlake High share the common theme by being relatively new high schools. Syracuse opened in 2007, Westlake opened in 2009.
The teams had played once, an 8-4 win for the Thunder back in 2012, but that's about where the similarities between the teams end now.
The Titans didn't celebrate after the win. Anderson added that his the team is trying not to get too high or too low emotionally — so it's on to the next round, another home game against Kearns at 4 p.m. Thursday.
You can reach prep sports reporter Patrick Carr at firstname.lastname@example.com. Follow him on Twitter at @patrickcarr_ or like him on Facebook at Facebook.com/patrickcarr17.
Sign up for e-mail news updates.


text:
The Long Realty Short Sale Resource Center provides you helpful information on short sales, the short sale process and how to search for short sale properties. Articles are provided by the Long Realty Short Sale Resource Group, a collaboration of Long Realty agents and managers with an expansive knowledge of short sales.
The "Short Sale" in a Market Recovery
June 4, 2013
While the housing market has shown strong signs of recovery over the last couple of years, many homeowners are still "underwater" and facing monthly financial difficulty in paying a debt that is greater than the value of their home. The decision to pursue a "Short Sale" is a difficult one and should be carefully considered with other options.
If considering a Short Sale, know that you are not alone. Zillow.com estimated that 42% of homeowners in Pima County had negative equity at the end of 2012. While that number decreases as values continue to rise, there are still many people that are "underwater". As the economy recovers and more, and higher paying jobs are created, many homeowners will be able to continue making their monthly payments and hopefully tip the scales back in favor of positive equity in their homes.
However, if you are struggling to make the payment and have negative equity, a Short Sale may be a good option for you. In a market recovery, Short Sales are beneficial for both the lending institutions and consumers with negative equity.
By successfully completing a Short Sale of your property, you may be able to limit the damage to your credit score (in comparison to a foreclosure) and you may also receive moving assistance funds from your mortgage holder (sometimes up to $3,000). Perhaps most importantly, depending on the type of loan you have, banks may often forgive the unpaid debt as part of a Short Sale agreement. Banks benefit as a Short Sale typically brings a higher sales price than a foreclosure. RealtyTrac.com estimated that in fourth quarter 2012, foreclosures sold at a 39% discount while Short Sales sold 23% below market. Neighborhoods benefit from higher sales prices and occupied, maintained homes. The seller can remain in the home until the Short Sale is completed.
As the housing market imploded and then eased into recovery, REALTORS®, consumers and lending institutions became more adept at processing Short Sales. Many banks now have quicker response times and require less financial documentation as part of this process. However, be prepared to present a comprehensive financial package to your lending institutions, including tax returns, pay stubs, bank statements, profit and loss statements and a letter explaining your circumstance, known as a "Hardship Letter".
If you are considering a Short Sale, consult with a REALTOR experienced in this field. It is a specialized category of real estate sales. An experienced Short Sale agent can guide you through this difficult process and help you through what is often times an emotional and exhaustive process. You should also seek legal and financial advice from a professional when considering selling your home "Short".
Short Sale Articles
December 12, 2013 Helpful Short Sale Tips and FAQ
The "Short Sale" in a Market Recovery
FHFA Announces New Short Sale Guidance
Many Homeowners Ask the Question: Foreclosure or Short Sale?
What's Happening in the Short Sale Market - June 2012
What's Happening in the Short Sale Market
Preparing for the Short Sale Listing Appointment
How Short Sales Affect the Market
Short Sale Resources
Before using this service, consider the following information.
Long Realty Company is not associated with the government, and our service is not approved by the government or your lender. Even if you accept this offer and use our service, your lender may not agree to change your loan. If you stop paying your mortgage, you could lose your home and damage your credit rating.


text:
Guy Pearce Is A First-Time Daddy!
Guy Pearce has become a father for the first time!
The Aussie actor made the exciting announcement on Twitter on Monday night, revealing that he and partner Carice van Houten had welcomed a little boy with a familiar name.
"A cute little package arrived and told us his name's Monte Pearce," the proud dad wrote. "We think we're gonna keep him. Placenta smoothie anyone?"
A cute little package arrived and told us his name's Monte Pearce. We think we're gonna keep him. Placenta smoothie anyone? @caricevhouten— Guy Pearce (@TheGuyPearce) August 29, 2016
Van Houten, who plays the Red Woman Melisandra in global hit TV series Game of Thrones, responded with a series of emojis that, to any mother, made perfect sense: A baby, a rollercoaster, a martini and a love heart.
👶🏼🎢😳😍🤒💩🍸🐣❤ https://t.co/vFQna7o4g1— Carice van Houten (@caricevhouten) August 29, 2016
Pearce, of course, famously starred in The Count Of Monte Cristo, prompting many to wonder if the new bub's name is an adorable throwback.
Considering how excited - and active - the couple have been on social media throughout the pregnancy, it's only a matter of time until we get a sweet snap of little Monte.
Almost don't fit the mirror anymore @TheGuyPearce .. pic.twitter.com/Qdx7FKW72g— Carice van Houten (@caricevhouten) July 13, 2016
Congratulations you two!


text:
This article is a guild information page for The Scarlet March of Moon Guard US.
The contents herein are entirely player made and in no way represent official World of Warcraft history or occurrences which are accurate for all realms. The characters and events listed are of an independent nature and applied for roleplaying, fictional, speculative, or opinions from a limited playerbase only. Guild pages must comply with the guild page policy.
The Scarlet March is a heavy role play guild in the realm Moon Guard and is role played as a religious organization dedicated to the eradication of the undead. They are a major adversary of the Scourge, and see little difference between the Forsaken and even the Death Knights of the Alliance.
General OverviewEdit
The Scarlet March is a branch of the Scarlet Crusade. The difference between the two is where the Crusade is simply concentrated on destroying all traces of the Undead and their Plague, the March is a little less narrow minded. The March not only dedicates itself to destroying all Undead but they are also attempting to rebuild the lives that were shattered by the Plague. Also the March wishes to try and preserve as much of their home's as they possibly can through means such as:
- Attempting to repopulate old towns
- Attempting to preserve many of the books found within places such as Stratholme
- Etc.
The Scarlet March holds to the same basic structure that the Scarlet Crusade has always used. Run much like a military would be run, the March expects the best from all of its followers.
The following is a list of the ranking system that the Scarlet March uses and a small description of each rank's responsibility mind you these are in no direct order:
- Scarlet Highlord
- High Inquisitor
- Scarlet Champion
- High Arcanist
- Scarlet Judge
- Chaplain
- Marksmen
- Knight
- Evoker
- Initiate
Scarlet HighlordEdit
- It's the Highlord's job to make sure that everyone does what needs to be done. He is the one who leads the entire March, and when it comes down to it he's the one that will make the final decision on anything done.
High InquisitorEdit
The High Inquisitor is in charge of the March's spiritual health. Most sermons and blessings come down from the Inquisitor. Also it's the Inquisitor who trains and commands the Chaplains.
Scarlet JudgeEdit
The Scarlet Judge is the strong arm of the March. In direct control of the military it's the Judge's job to distribute the Knights and Marksmen to where they need to be in order for our cities to be safe and to kill the undead more effectively.
High ArcanistEdit
The High Arcanist is the master of the Arcane. He bends magic to his will with a simple flick of his fingers and his knowledge is unparalleled. It's for that reason that the High Arcanist and his/hers Evokers are in charge of keeping the records of the March, restoring lost books, and creating new ways of destroying the Scourge.
Scarlet ChampionEdit
Only the most elite fighters and most devout patrons of the Holy Light can call themselves a Scarlet Champion. This small yet powerful group is the Highlord's handpicked guard and get their orders directly from him.
These devoted men and women have given everything for the Light they are the heart of the Scarlet March. When not on the field healing the soldiers and smiting down the Undead they are within the cities talking with the people. It's the Chaplain's who are responsible for bringing in more followers into the March, and they are the ones who will bless it with the strength and wisdom of the Holy Light.
Marksman & KnightEdit
These are the brave men and women who guard our gates and march against our enemies. They take their orders from the Scarlet Judge and destroy the Scourge with no mercy while protecting the innocents from the evil the Lich King has brought upon us.
Spellcasters of all sorts; whether they study fire, ice, or the arcane these men and women study under the High Arcanist in order to better themselves. They also go into battle with the army and help the High Arcanist with his studies.
The entry phase of the Scarlet March. This is the first step you take before deciding which path through the March you wish to take.
Race, Class, & Level RestrictionsEdit
Humans are always welcomed with open arms. Dwarves will find little trouble joining as well. Only Gnomes who show us that they truly believe in our cause and within the teachings of the light may join. Night Elves are welcome to stay for shelter but will be politely turned away if they inquire to join. Dreanei are the same as Night Elves. Although none can question the Dreanei faith their battle is with the Burning Legion, not the Scourge.
No Death Knights period. I don't care if you say you're alive or not...you literally have the stench of the Scourge all over you and although any soldier of The March will hold his blade while in Varian's land...I can't say the same for outside of his borders.
Warlocks have to be EXTREMELY good at hiding their association with demons and be able to fool the High Arcanist that you're a mage.
No Night Elves...so no Druids.
No Dreanei...so no Shaman.
Priest, Paladin, Warrior, Rogue, Hunter, Mage are all openly welcome.
There is no level restriction, however be aware that your level 1-10 alt will not clog space up in the guild.
If you're level 30+ then you're the perfect level because we can start getting the appropriate gear for you to roleplay in.
60+ is a fantastic level because you can help lower levels out with gear and it's a lot easier to get around with more higher leveled characters.


text:
VIENNA (AP) — A senior U.S. envoy accused Iran of "deception, defiance and delay" Wednesday in dealing with international concerns about its nuclear program, reflecting frustration over Tehran's expanding uranium enrichment program and stalled U.N. attempts to determine whether Tehran has worked secretly on nuclear arms.
Joseph Macmanus, the chief U.S. delegate to the International Atomic Energy Agency,also suggested that the U.S. might push for tougher diplomatic action in the coming months.
While not going into details, his comments indicated that America might lobby the IAEA board to ask for a special inspection of Parcin, a facility that the agency suspects was used to test explosive triggers for a nuclear weapon, or that the United States would seek an IAEA resolution critical of Tehran.
International criticism of Iran has been relatively muted since last week's nuclear talks in which Tehran showed interest in proposals made by the United States and five other world powers. While expressing concern about enrichment and the deadlocked probe, the six powers avoided tough language and mentioned the "useful meetings" that produced the proposals in a joint statement Tuesday to the 35-nation International Atomic Energy Agency board.
By contrast, the comments Wednesday to the same meeting by Macmanus were unusually hard edged, suggesting they were meant to signal that pressure on Iran over its nuclear activities would not diminish.
Without having to pay heed to Russia and China — countries in the six-power group that are traditionally softer on Iran than Washington — Macmanus concentrated on expressing the U.S. view of Iran's alleged failure to meet its international obligations and diminish concerns that it wants nuclear weapons.
Iran denies any such aspirations. But it hid its enrichment program for years and is rapidly expanding it, prompting suspicions that it was less interested in using it to make reactor fuel and more in its other use — producing fissile warhead material. The IAEA also suspects that Tehran worked secretly on nuclear weapons, basing its assessment mostly on intelligence from the U.S., Israel and West European nations. Tehran says the intelligence is faked and refuses to allow the IAEA to resume a probe of the allegations until details of how that should proceed are worked out — a stipulation the West dismisses as a delaying tactic.
Macmanus, in comments to the closed meeting made available to media, focused on both Iran's expanding enrichment program and refusal to allow IAEA experts access to sites, officials and documents it wants to probe in its investigations of Parcin and other suspicions of nuclear weapons work.
"We are deeply concerned with what appears to be Iran's unwavering commitment to deception defiance and delay," he said. "Iran ... has chosen to take further provocative actions."
Asked about possible requests for a special inspection or an IAEA board resolution in the future, he later told reporters that "some adjustment might have to be made" in ways to address concerns about Iran, adding that will be taken up by the board "over the next several months."
Iran can refuse a special inspection but that would set it up for referral to the U.N. Security Council. Like others before it, a resolution critical of Iran also would go automatically to the council, adding diplomatic pressure on Tehran.
Diplomats say the proposal made to Iran late last month by the United States, Russia, China, Britain, France and Germany, would obligate Iran to decommission its centrifuge plant at Fordo now making higher-enriched uranium and ship out the approximately 165 kilograms (about 365 pounds) it now has as well as allowing increased U.N. oversight.
With no such material stored and none being made, Iran's most direct path to quick manufacture of weapons-grade uranium would be eliminated — about 250 kilograms of higher-enriched uranium are needed to be able to have enough material for one nuclear bomb with further enrichment .
In return, the six are offering to help supply and run Iran's research reactor which is fueled by plates made from higher enriched uranium, coupled with what Iran wants most — relief from sanctions meant to penalize Iran for refusing to heed U.N. Security Council demands to stop all enrichment.


text:
Grand Rapids Open House
WMU-Cooley Law School
Thursday, November 15, 2012 from 4:00 PM to 7:00 PM (EST)
Event Details
Get started! Attend the Grand Rapids Open House. Meet with admissions, students, faculty, and alumni.
Register online now or onsite the day of the Open House.
» Learn about the admissions process and applying to law school.
» Learn about financial aid and career services.
» Earn up to 85% of tuition with an Honors Scholarship.
» Enjoy a mock class experience.
» Take a campus tour.
Call toll free (800) 874-3511 or email firstname.lastname@example.com.
When & Where
Cooley Law School Grand Rapids Campus
111 Commerce Ave SW
Grand Rapids,
Thursday, November 15, 2012 from 4:00 PM to 7:00 PM (EST)
Add to my calendar
WMU-Cooley Law School
Western Michigan University Cooley Law School combines the status of a nationally-ranked, public, comprehensive research university with the commitment to practical legal education of an independent, non-profit, national law school. Since 1972, nearly 20,000 graduates have gone on to become leaders in every state in the nation and around the world, in the legal profession, in public service and in private commerce.


text:
Animal Species:Bombardier Beetle
The Bombardier Beetle belongs to the ground beetle family (Carabidae), of which there are about 2,500 species in Australia.
Standard Common Name
Bombardier Beetle
Number of species
Size range
1.2 cm - 1.8 cm
The Bombardier Beetle is found in eastern Australia.
The Bombardier Beetle lives on undisturbed riverbanks and floodplains.
Feeding and Diet
Most Bombardier Beetles are carnivorous and hunt on the ground or in trees. The larvae feed on other insects.
Other behaviours and adaptations
When threatened, the Bombardier Beetle uses a special gland at its rear end to mix together two chemicals, resulting in an explosion with a loud popping noise. The noise alone is usually enough to startle a would-be predator and, if not, the accompanying sizzle of steam and spray of chemicals from the beetle's behind does the trick! The beetle produces temperatures of over 100 oC in the gland, which acts as a specialised, reinforced explosion chamber. Up to 80 explosions may be produced over a four-minute period.
- Species:
- verticalis
- Genus:
- Pheropsophus
- Subfamily:
- Brachininae
- Family:
- Carabidae
- Superfamily:
- Caraboidea
- Suborder:
- Adephaga
- Order:
- Coleoptera
- Class:
- Insecta
- Subphylum:
- Uniramia
- Phylum:
- Arthopoda
- Kingdom:
- Animalia
Last Updated:
Tags beetles, insects, arthropods, invertebrates, identification, wildlife of sydney,


text:
Darrell Gulin was born and raised in Seattle, Washington and earned a Business degree from the University of Washington. Gulin has been photographing for over 23 years and has been actively marketing his work over the last 21 years.
© 2018 Canon U.S.A., Inc. All Rights Reserved. Reproduction in whole or in part without permission is prohibited.


text:
This image was useful to her throughout many twists and turns of her journey with cancer, reminding her to stay focused on where she wanted to go and not let herself get lost in her fears. Patricia did "make it", and she counts guided imagery as one of the tools that enabled her to do so.
When, like Patricia, you are diagnosed with cancer, you can find yourself over-whelmed with emotions at a time when you most need to keep your wits about you. While you are alive, you have hope, and you have options. You have will, imagination, and powerful natural healing abilities within you that you can stimulate by the way you use your mind.
What Is Guided Imagery?
Guided Imagery like the kind Patricia used trains your mind to listen to your body's wisdom. "Imagery" is simply a flow of thoughts you can see, hear, feel, smell or taste. It is the expression of dreams and daydreams; memories and reminiscence; plans, projections and possibilities. It is the language of the arts, the emotions, and most important, of the deeper self.
"Guided imagery" is a process of being lead or "guided" to tune in to your own natural imagery, and to "listen" to the messages and meanings that imagery may reveal about what you need to heal. Guided imagery techniques range from simple visualization and direct imagery-based suggestion, to metaphor, dialogue and story-telling.
Why Use Guided Imagery to Fight Cancer?
The shock, disorientation and anxiety that often come with a serious cancer diagnosis can overwhelm one's sense of confidence and make it hard for you to feel effective or powerful in a time when you may most need inner strength. Imagery can help you reconnect with your own resources and begin to use them effectively on your own behalf. Guided imagery CDs are easy to use, inexpensive, and have rapid psychological benefits.
Research at the Carnegie-Mellon Institute in Pittsburgh has shown this method to be extremely effective in helping people shift from one mood state to another. Learning to shift from helplessness to hopefulness at will is an empowering experience for anyone, especially for anyone feeling overwhelmed with the fear of cancer.
The benefits of imagery are tangible and physical - studies prove that patients who use guided imagery can experience an increase the numbers and aggressiveness of natural killer cells when practiced over time, a reduction in complications from surgery, relief from pain, and can experience a lessening of the adverse effects of chemotherapy.
How Guided Imagery Helps
When diagnosed with cancer, you may not only face an on-going threat to life and well-being, you often also face conflicting recommendations on the part of the best experts in the field and the resulting uncertainty as to the best course of treatment. One use of guided imagery CDs is to help you make good decisions about your treatment. For instance, by using guided imagery to help you imagine a wise and caring figure, called an "inner advisor" or "inner wisdom" that has your best interests at heart, you can make the best possible decisions in a time of stress.