text:
The posts on this site are strictly my own, and do not necessarily represent the views, opinions etc of Red Hat.
Building Grafana from source on Fedora
Here are the official docs for building Grafana from source. And below are my notes on how to build Grafana, starting from a clean Fedora 27 Cloud image.
# Install Dependencies curl https://dl.yarnpkg.com/rpm/yarn.repo > /etc/yum.repos.d/yarn.repo sudo yum install golang yarn rubygems ruby-devel redhat-rpm-config rpm-build git -y gem install fpm sudo yarn install --pure-lockfile npm install -g yarn & yarn install
Setup the go environment.
# go environment mkdir ~/go export GOPATH=~/go export PATH=$PATH:$(go env GOPATH)/bin
Download the various repositories required to build. Here you could also clone your fork/branch of Grafana into $GOPATH/src.
# Pull sources required to build go get github.com/grafana/grafana golang.org/x/sync/errgroup github.com/codegangsta/cli cd $GOPATH/src/github.com/grafana/grafana npm install
Now you can make any sort of local changes, or just build from HEAD.
# go run build.go setup # takes 45 seconds $ time go run build.go build pkg-rpm # takes about 7 minutes
The build will spit out an RPM in a folder called dist:
Created package {:path=>"./dist/grafana-5.0.0-1517715437pre1.x86_64.rpm"}
Docker operations slowing down on AWS (this time it's not DNS)
I'm CC'd on mails when things get slow, but never when things work as expected or are fast…oh well. Like an umpire in baseball, if we are doing our jobs, we are invisible.
Subject: Docker operations slowing down
I reach for my trusty haiku for this type of thing:
Ah but in this scenario, it is something more…sinister (my word). What could be more sinister than DNS, you say? It's the magical QoS system by which a cloud provider creatively rents you resources. The system that allows for any hand-wavy repackaging of compute or network or disk into a brand new tier of service…
Platinum. No, super Platinum. What's higher than platinum? Who cares, we are printing money and customers love us because we have broken through their antiquated finance process. We will gladly overpay via OpEx just to avoid that circus.
But I digress…
In this scenario, I was working with one of our container teams folks who had a report of CI jobs failing and someone had debugged a bit and pinned the blame on docker. I watch the reproducer run. It is running
docker run --rm fedora date
in a tight loop. I watch as docker daemon gets through its 5000th loop iteration, and…still good to go. On average, ~3 seconds to start a container and delete it. Not too bad, certainly not something that a CI job shouldn't be able to handle. I continue to stare at tmux and then it happens…WHAM! 82 seconds to start the last container. Ahh, good. Getting a reproducer is almost always the hardest part of the process. Once we have a tight debug loop, smart people can figure things out relatively quickly.
I am looking at top in another window, and I see systemd-udev at the top of the list…what the…
As much as I would love to blame DNS for this, I have a hunch this is storage related now, because the reproducer shouldn't be doing anything on the network. Now I am running ps in a loop and grepping for " D ". Why? Because that is the process state when a thread is waiting on I/O. I know this because of several terribly painful debugging efforts with multipath in 2010. Looking back, it may have been those situations that have made me run screaming from filesystem and disk performance issues ever since 🙂
From man ps:
PROCESS STATE CODES Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process: D uninterruptible sleep (usually IO) R running or runnable (on run queue) S interruptible sleep (waiting for an event to complete) T stopped by job control signal t stopped by debugger during the tracing W paging (not valid since the 2.6.xx kernel) X dead (should never be seen) Z defunct ("zombie") process, terminated but not reaped by its parent
Normally, processes oscillate between R and S, often imperceptibly (well, at least not something you see very often in top). You can easily trace this with the systemtap script sleepingBeauties.stp if you really need to. This script will print a backtrace of any thread that enters D state for a configurable amount of time.
Anyway here are the threads that are in D state.
root 426 0.4 0.0 0 0 ? D 16:10 0:08 [kworker/7:0] root 5298 0.2 0.0 47132 3916 ? D 16:39 0:00 /usr/lib/systemd/systemd-udevd root 5668 0.0 0.0 47132 3496 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 24112 0.5 0.0 0 0 ? D 16:13 0:08 [kworker/u30:0] root 5668 0.0 0.0 47132 3832 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 5656 0.0 0.0 47132 3884 ? D 16:39 0:00 /usr/lib/systemd/systemd-udevd root 29884 1.1 0.0 0 0 ? D 15:45 0:37 [kworker/u30:2] root 5888 0.0 0.0 47132 3884 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 5888 0.5 0.0 47132 3904 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 5964 0.0 0.0 47132 3816 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 29884 1.1 0.0 0 0 ? D 15:45 0:37 [kworker/u30:2] root 5964 0.3 0.0 47132 3916 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 5964 0.2 0.0 47132 3916 ? D 16:40 0:00 /usr/lib/systemd/systemd-udevd root 24112 0.5 0.0 0 0 ? D 16:13 0:08 [kworker/u30:0]
That is interesting to me. udevd is in the kernel's path for allocate/de-allocate storage devices. I am now convinced it is storage. kworker is a workqueue kernel thread that fires when the kernel's writeback watermarks (dirty pages) are hit. For my extreme low latency work, I documented how to shove these in a corner in my Low Latency Tuning Guide for Red Hat Enterprise Linux 7.
I move over to another tmux pane and I try:
dd if=/dev/zero of=/root/50MB bs=1M count=10 oflag=sync
I know that if this does not complete in < 5 seconds, something is terribly hosed. Aaaaaand it hangs. This process now shows up in my ps loop looking for D state processes. So I have it narrowed down. Something is wrong with the storage on this VM, and it only shows up after 5000 containers are started (well, I am told it varies by a few thousand here and there).
This may seem like a tangent but I promise it is going somewhere:
Nearly two years ago, when we were first standing up openshift.com version 3 on AWS, we ran into a few eerily similar issues. I remember that our etcd cluster would suddenly start freaking out (that is a technical term). Leader elections, nodes going totally offline…And I remember working with our AWS contacts to figure it out. At the time it was a little less well-known, and today just by googling it appears fairly well understood. The issue with this reproducer turns out to be something called a BurstBalance. BurstBalance is AWS business logic interfering with all that is good and holy. If you purchase storage, you should be able to read and write from it, no?
As with all public cloud, you can do whatever you want…for a price. BurstBalance is the creation of folks who want you to get hooked on great performance (gp2 can run at 3000+ IOPS), but then when you start doing something more than dev/test and run into these weird issues, you're already hooked and you have no choice but to pay more for a service that is actually usable. This model is seen throughout public cloud. For example, take the preemptible instances on GCE or the t2 instance family on AWS.
I have setup my little collectd->graphite->grafana dashboard that I use for this sort of thing. You can see things are humming along quite nicely for a while, and then…yeah.
Once the reproducer exhausts the gp2 volume's BurstBalance, things go very, very badly. Why? Simple. Applications were not written to assume that storage would ever slow down like this. Issues in docker cascade back up the stack until finally a user complains that it took 5 minutes to start their pod.
The reason is that we have not paid our bounty to the cloud gods.
Here is BurstBalance and the magical AWS QoS/business logic in action.
You can see it looks a lot like my grafana graphs…quota is exhausted, and the IOPS drop to a trickle.
What would happen then if we did kneel at the alter of Bezos and pay him his tithe? I will show you.
The reproducer is chugging along, until it slams into that magical AWS business logic. Some QoS system somewhere jumps for joy at the thought of earning even more money. This time, we will pay him his fee…for science.
You can see that our reproducer recovers (lower is better) once we flip the volume type to provisioned IOPS (io1)…this was done on the fly. We set the io1 volume to 1000 IOPS (mostly random choice…) which is why it is slightly higher after the recovery than it was before the issue occurred. gp2 can crank along really, really fast. That is, until…
The take aways from this debugging session are:
- Regardless of cloud provider, you pay a premium for both performance and determinism.
- If you think you are saving money up front, just wait until the production issues start rolling in which, conveniently, can easily be solved by simply clicking a little button and upgrading to the next tier. Actually, it is brilliant and I would do the same if I had the unicorn QoS system at my disposal, and was tasked with converting that QoS system into revenue.
- I now must proactively monitor BurstBalance and flip volumes to io1 instead of let them hit the wall in production. Monitoring for this (per AWS documentation, use CloudWatch)
is an additional feeappears to be included in their CloudWatch free tier.
- Perhaps we flip all volumes to io1 proactively and then flip them back when the critical period is over.
- One thing I ran out of time to verify is what happens to my BurstBalance if I flip to io1, then back to gp2? Is my BurstBalance reset? Probably not, but I haven't done the leg work yet to verify.
- We will do less I/O when using overlay2 (might just delay the inevitable).
- All super critical things (like etcd) get io1 out of the gate. No funny business.
- Incidentally, we spent an inordinate amount of time with etcd I/O lately. The fruit of that effort is posted in the OpenShift Scaling and Performance guide etcd section.
Juggling backing disks for docker on RHEL7, using atomic storage migrate
Quick article on how to use the atomic storage commands to swap out an underlying storage device used for docker's graph storage.
- I am currently using overlay2 for docker storage, and /var/lib/docker is currently on my root partition
- I want to add a 2nd disk just for docker storage.
- I want to keep my images, rather than have to download them again.
I have a few images in my system:
# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/openshift/hello-openshift latest 305f93951299 3 weeks ago 5.635 MB docker.io/centos centos7 3bee3060bfc8 6 weeks ago 192.6 MB docker.io/monitoringartist/grafana-xxl latest 5a73d8e5f278 10 weeks ago 393.4 MB docker.io/fedora latest 4daa661b467f 3 months ago 230.6 MB docker.io/jeremyeder/c7perf latest 3bb51319f973 4 months ago 1.445 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools latest 264d7d025911 4 months ago 1.488 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7 latest 41a4953dbf95 4 months ago 192.5 MB docker.io/busybox latest 7968321274dc 6 months ago 1.11 MB # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-root 193G 162G 23G 88% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 804K 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vdc1 100G 33M 100G 1% /var/lib/docker/overlay /dev/vda1 2.0G 549M 1.5G 28% /boot
All of docker's storage right now consumes about 4GB. It's important to verify this because the migrate commands we're about to walk through require this much space to complete the migration:
# du -hs /var/lib/docker 3.9G /var/lib/docker
By default, the atomic migrate commands will write to /var/lib/atomic, so whatever filesystem holds that directory will need at least (in my case) 4GB free.
The migration process has several phases:
- Export any containers and images.
- Allow user to adjust storage on the system.
- Allow user to adjust storage configuration of docker.
- Import containers and images back into the new docker graph storage.
I'm using a VM with spinning disks so this takes a little longer than it otherwise might, but let's start the export:
# time atomic storage export Exporting image: 5a73d8e5f278 Exporting image: 3bb51319f973 Exporting image: 7968321274dc Exporting image: 3bee3060bfc8 Exporting image: 4daa661b467f Exporting image: 264d7d025911 Exporting image: 41a4953dbf95 Exporting image: 305f93951299 Exporting volumes atomic export completed successfully real 1m57.159s user 0m1.094s sys 0m6.190s
OK that went oddly smoothly, let's see what it actually did:
# find /var/lib/atomic/migrate /var/lib/atomic/migrate /var/lib/atomic/migrate/info.txt /var/lib/atomic/migrate/containers /var/lib/atomic/migrate/images /var/lib/atomic/migrate/images/4daa661b467f23f983163d75f0b87744cd3d88a2aed11be813d802606e8f13df /var/lib/atomic/migrate/images/3bee3060bfc81c061ce7069df35ce090593bda584d4ef464bc0f38086c11371d /var/lib/atomic/migrate/images/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768 /var/lib/atomic/migrate/images/264d7d0259119cf980fb95759865938765ccb3f1aa24600cbac49bea6b5b8cfb /var/lib/atomic/migrate/images/305f939512995147aa964bceef36a4a83226fae523c52b011fd69c9a229e3460 /var/lib/atomic/migrate/images/5a73d8e5f27861df210b03ca872530b6ab8b20b6a0d9c815022da3e0812df089 /var/lib/atomic/migrate/images/3bb51319f9734038d7b2d3c67cae6c25bbd9df18163cd7810ffcff952cbe0608 /var/lib/atomic/migrate/images/41a4953dbf957cfc562935239a3153a5da6101f32fa30da7b4a506f23cfcde9d /var/lib/atomic/migrate/volumes /var/lib/atomic/migrate/volumes/volumeData.tar.gz
Seems reasonable…incidentally that info.txt just includes the name of the storage driver used at the time migrate was executed.
# du -hs /var/lib/atomic 3.8G /var/lib/atomic
OK let's do the deed:
# atomic storage reset Docker daemon must be stopped before resetting storage
Oh, I guess that would make sense.
# systemctl stop docker # atomic storage reset
OK, at this point /etc/sysconfig/docker-storage is reset to it's default state, and I have nothing in my docker graph storage.
Because I want to continue to use overlay2, I will use the atomic storage modify command to make that so:
# atomic storage modify --driver overlay2 # cat /etc/sysconfig/docker-storage DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "
Things are looking good so far.
Now about adding more storage.
- I have added a new virtual storage device to my VM called /dev/vdc1
- I have partitioned and formatted it with XFS filesystem.
- I have mounted it at /var/lib/docker and setup an fstab entry.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 200G 0 disk ├─vda1 252:1 0 2G 0 part /boot └─vda2 252:2 0 198G 0 part ├─vg0-swap 253:0 0 2G 0 lvm [SWAP] └─vg0-root 253:1 0 196.1G 0 lvm / vdb 252:16 0 100G 0 disk └─vdb1 252:17 0 100G 0 part vdc 252:32 0 100G 0 disk └─vdc1 252:33 0 100G 0 part /var/lib/docker
At this point we are ready to restart docker and import the images from my previous storage. First let me verify that it's OK.
# systemctl start docker # docker info|grep -i overlay2 Storage Driver: overlay2
Cool, so docker started up correctly and it has the overlay2 storage driver that I told it to use with the atomic storage modify command (from previous step).
Now for the import…
# time atomic storage import Importing image: 4daa661b467f ae934834014c: Loading layer [==================================================>] 240.3 MB/240.3 MB Loaded image: docker.io/fedora:latest Importing image: 3bee3060bfc8 dc1e2dcdc7b6: Loading layer [==================================================>] 200.2 MB/200.2 MB Loaded image: docker.io/centos:centos7 Importing image: 7968321274dc 38ac8d0f5bb3: Loading layer [==================================================>] 1.312 MB/1.312 MB Loaded image: docker.io/busybox:latest Importing image: 264d7d025911 827264d42df6: Loading layer [==================================================>] 202.3 MB/202.3 MB 9ca8c628d8e7: Loading layer [==================================================>] 10.24 kB/10.24 kB a03f55f719da: Loading layer [==================================================>] 1.336 GB/1.336 GB Loaded image: brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools:latest Importing image: 305f93951299 5f70bf18a086: Loading layer [==================================================>] 1.024 kB/1.024 kB c618fb2630cb: Loading layer [==================================================>] 5.637 MB/5.637 MB Loaded image: docker.io/openshift/hello-openshift:latest Importing image: 5a73d8e5f278 8d4d1ab5ff74: Loading layer [==================================================>] 129.4 MB/129.4 MB 405d1c3227e0: Loading layer [==================================================>] 3.072 kB/3.072 kB 048845c41855: Loading layer [==================================================>] 277.2 MB/277.2 MB Loaded image: docker.io/monitoringartist/grafana-xxl:latest Importing image: 3bb51319f973 34e7b85d83e4: Loading layer [==================================================>] 199.9 MB/199.9 MB ab7578fbc6c6: Loading layer [==================================================>] 3.072 kB/3.072 kB 3e89505f5573: Loading layer [==================================================>] 58.92 MB/58.92 MB 753668c55633: Loading layer [==================================================>] 1.169 GB/1.169 GB d778d7335b8f: Loading layer [==================================================>] 11.98 MB/11.98 MB 5cd21edffb34: Loading layer [==================================================>] 45.1 MB/45.1 MB Loaded image: docker.io/jeremyeder/c7perf:latest Importing image: 41a4953dbf95 Loaded image: brew-pulp-docker01.redacted.redhat.com:8888/rhel7:latest Importing volumes atomic import completed successfully Would you like to cleanup (rm -rf /var/lib/atomic/migrate) the temporary directory [y/N]n Please restart docker daemon for the changes to take effect real 1m23.951s user 0m1.391s sys 0m4.095s
Again went smoothly. I opted not to have it automatically clean up /var/lib/atomic/migrate automatically because I want to verify a thing or two first.
Let's see what's on my new disk:
# df -h /var/lib/docker Filesystem Size Used Avail Use% Mounted on /dev/vdc1 100G 3.9G 97G 4% /var/lib/docker
OK that looks reasonable. Let's start docker and see if things imported correctly:
# systemctl restart docker # docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/openshift/hello-openshift latest 305f93951299 3 weeks ago 5.635 MB docker.io/centos centos7 3bee3060bfc8 6 weeks ago 192.6 MB docker.io/monitoringartist/grafana-xxl latest 5a73d8e5f278 10 weeks ago 393.4 MB docker.io/fedora latest 4daa661b467f 3 months ago 230.6 MB docker.io/jeremyeder/c7perf latest 3bb51319f973 4 months ago 1.445 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools latest 264d7d025911 4 months ago 1.488 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7 latest 41a4953dbf95 4 months ago 192.5 MB docker.io/busybox latest 7968321274dc 6 months ago 1.11 MB
Images are there. Can I run one?
# docker run --rm fedora pwd /
Indeed I can. All seems well.
This utility is very handy in scenarios where you want to do some surgery on the backend storage, but do not want to throw away/download images and containers. I could envision using this utility when
- Moving from one graph driver to another. Note that we have SELinux support coming to overlay2 in RHEL 7.4.
- Perhaps you have a lot of images or containers and slow internet.
Either way, this process was about as smooth as it could be…and a very clean UX, too.
Building KDAB hotspot 'perf' visualization tool on Fedora
As any respectable software performance person knows, perf is your best (only?) friend. For example, perf report -g has shined a light into the deepest, darkest corners of debugging territory. Since you asked, it can happily run in a container, too (albeit requiring elevated privileges, but we're debugging here…).
Typically console formatted output is fine for grokking perf reports, but having recently become addicted to go's pprof visualization (dot format), handy flame graphs, and on the morbid occassion, VTune, I started looking around for a way to more clearly understand a particular perf recording.
Googling turned up an interesting QT-based tool called hotspot by a company called KDAB. Screenshots indicate it might be worth kicking the tires.
After some bouncing around figuring out Fedora equivalent package names, I was able to quickly build and run hotspot. I ran a quick perf record to see if it was going to work at all:
$ sudo perf record --call-graph dwarf sleep 10 $ ./bin/hotspot ./perf.data
And voila…
Folks at KDAB even included a built-in flame graph:
Interface is clean, bug-free and useful. Trying to load a large perf.data file was a bit ugly and RAM-intensive; I would likely stick to command-line parsing for those. Or, as we do in pbench, reduce the collection frequency to 100Hz and take bite-sized samples over the life of the test.
nsinit: per-container resource monitoring of Docker containers on RHEL/Fedora
The use-case for per-application resource counters
Administrators of *NIX-based systems are quite accustomed to viewing resource counters strewn throughout the system, in places like /proc, /sys and more recently /cgroup or /sys/fs/cgroup. With the release of RHEL6 came widespread enterprise adoption of Control Groups (cgroups), which had been implemented steadily over a series of years, and vetted both there as well as in Fedora (RHEL's upstream).
Implementing cgroups not only let sysadmins carve up a single OS into multiple logical partitions, it also bought them per-cgroup counters that the kernel maintains. That's in addition to common use-cases such as quality of service guarantees or charge-back.
Docker's unique twist
With the recent uptick in adoption of Linux containers (Docker encapsulates several mature technologies into an impressive usability package), administrators might be wondering where the per-container resource counters are. We're in luck! Since Docker heavily relies on Cgroups, many of the counters that sysadmins are familiar with "just work". They could benefit from some usability improvements, but if you're comfortable spelunking through the cgroup VFS, you can dig them out fairly easily.
I should note that the specific hierarchy and commands below are specific to RHEL and Fedora, so you might have to customize some paths or package names for your system.
In the most recent versions of Fedora, engineers have begun building and shipping a binary called 'nsinit', which is part of libcontainer, which is the "execution driver" for Docker. nsinit is a very powerful debugging utility that lets sysadmins not only view per-container resource counters, but also view the container's runtime configuration and "jump into" a running container.
How to use the nsinit utility
First you should grab a copy from Fedora, or build it yourself. Building it yourself is an unnecessarily complicated exercise; so I'm glad they started building it for Fedora so you can just do:
# yum install --enablerepo=updates-testing golang-github-docker-libcontainer $ rpm -qf `which nsinit` golang-github-docker-libcontainer-1.1.0-7.git29363e2.fc20.x86_64 # nsinit NAME: nsinit - A new cli application USAGE: nsinit [global options] command [command options] [arguments...] VERSION: 0.1 COMMANDS: exec execute a new command inside a container init runs the init process inside the namespace stats display statistics for the container config display the container configuration nsenter init process for entering an existing namespace pause pause the container's processes unpause unpause the container's processes help, h Shows a list of commands or help for one command
I'll cover the most useful of nsinit's capabilities; config, stats and exec.
Note: nsinit currently requires that you run it while you're inside the container's state directory. So from here on, all commands assume you're in there.
So, something like this:
# docker ps -q 4caad549289 # CID=`docker ps -q` # cd /var/lib/docker/execdriver/native/$CID* # ll total 8 -rw-r-xr-x. 1 root root 3826 Sep 1 20:11 container.json -rw-r--r--. 1 root root 114 Sep 1 20:11 state.json
Those files are plain-text readable, although not very human-readable. nsinit pretty-prints these files. For example, an abridged verison of the output of nsinit config (full version here). Note that you can get much of this info (but not all) from docker inspect.
# nsinit config { "mount_config": { "mounts": [ { "type": "bind", "source": "/var/lib/docker/init/dockerinit-1.1.1", "destination": "/.dockerinit", "private": true }, { "type": "bind", "source": "/etc/resolv.conf", "destination": "/etc/resolv.conf", "private": true }, <snip> "mount_label": "system_u:object_r:svirt_sandbox_file_t:s0:c631,c744" }, "hostname": "4caad5492898", "environment": [ "HOME=/", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/goroot/bin:/gopath/bin", "HOSTNAME=4caad5492898", "DEBIAN_FRONTEND=noninteractive", "GOROOT=/goroot", "GOPATH=/gopath" ], "namespaces": { "NEWIPC": true, "NEWNET": true, "NEWNS": true, "NEWPID": true, "NEWUTS": true }, "capabilities": [ "CHOWN", "DAC_OVERRIDE", "FOWNER", "MKNOD", "NET_RAW", "SETGID", "SETUID", "SETFCAP", "SETPCAP", "NET_BIND_SERVICE", "SYS_CHROOT", "KILL" ], "networks": [ { "type": "loopback", "address": "127.0.0.1/0", "gateway": "localhost", "mtu": 1500 }, { "type": "veth", "bridge": "docker0", "veth_prefix": "veth", "address": "172.17.0.6/16", "gateway": "172.17.42.1", "mtu": 1500 } ], "cgroups": { "name": "4caad5492898f1a4230353de15e2acfc05809c69d05ec7289c6a14ef6d57b195", "parent": "docker", "allowed_devices": [ <snip> "process_label": "system_u:system_r:svirt_lxc_net_t:s0:c631,c744", "restrict_sys": true }
The stats mode is far more interesting. nsinit reads cgroup counters for CPU and memory usage. The network statistics come from /sys/class/net/<EthInterface>/statistics. From here you can see how much memory your application is using, chart it's growth, watch CPU utilization, cross-check data from other tools, etc.
{ "network_stats": { "rx_bytes": 180568, "rx_packets": 89, "tx_bytes": 28316, "tx_packets": 92 }, "cgroup_stats": { "cpu_stats": { "cpu_usage": { "total_usage": 985559718, "percpu_usage": [ 43613750, 79789656, 132486590, 78759739, 49063680, 60703059, 36277458, 35919550, 36329424, 20096103, 8148695, 25279255, 0, 0, 0, 6144761, 14814784, 2612915, 95162480, 33853872, 114861235, 71115914, 6533416, 33993382 ], "usage_in_kernelmode": 510000000, "usage_in_usermode": 440000000 }, "throlling_data": {} }, "memory_stats": { "usage": 27992064, "max_usage": 29020160, "stats": { "active_anon": 4411392, "active_file": 3149824, "cache": 22278144, "hierarchical_memory_limit": 9223372036854775807, "hierarchical_memsw_limit": 9223372036854775807, "inactive_anon": 0, "inactive_file": 19128320, "mapped_file": 3723264, "pgfault": 94783, "pgmajfault": 25, "pgpgin": 19919, "pgpgout": 13902, "rss": 4460544, "rss_huge": 2097152, "swap": 0, "total_active_anon": 4411392, "total_active_file": 3149824, "total_cache": 22278144, "total_inactive_anon": 0, "total_inactive_file": 19128320, "total_mapped_file": 3723264, "total_pgfault": 94783, "total_pgmajfault": 25, "total_pgpgin": 19919, "total_pgpgout": 13902, "total_rss": 4460544, "total_rss_huge": 2097152, "total_swap": 0, "total_unevictable": 0, "unevictable": 0 }, "failcnt": 0 }, "blkio_stats": {} } }
nsenter is commonly used to run a command inside an existing container, something like
# nsenter -m -u -n -i -p -t 19119 bash
Where 19119 is the PID of a process in the container. Ugly. nsinit makes this slightly easier (at least IMHO):
# nsinit exec cat /etc/hostname 4caad549289 # nsinit exec bash bash-4.2# exit
nsinit's capabilities and reported statistics are incredibly useful when debugging the implementation of QoS for each container, implementing/verifying resource-ceilings/guarantees, and for a more complete understanding of what your containers are doing.
This area is fast-moving…I did want to call out two other important developments, which should ultimately have more broad applicability than nsinit.
Google has published a project called cAdvisor that provides a basic web interface, but more importantly an API for higher layers (such as Kubernetes) to use.
Red Hat has proposed container support for Performance Co-Pilot, a system-level performance monitoring utility in RHEL7, along with goals of teaching many other tools about containers.
Using SCHED_FIFO in Docker containers on RHEL
Well, I've been asked about this quite a few times now, so I figured a blog post was in order…
When I was trying to get cyclictest running in a container, I ran into a little snag. I couldn't run realtime prio tasks inside a container by default. I checked all the normal ulimit stuff for RT, but no dice. But I did find a way (ugly).
If you do want to run SCHED_FIFO tasks you can in fact do so, like this:
Run a privileged container (because of cap_sys_nice being dropped by docker) adding this to your docker run command:
Or, if you have a more recent version of Docker, add this to your docker run command:
Set rt_runtime_us > 0 for the parent cgroup of where docker containers end up in the heirarchy:
# echo 950000 > /sys/fs/cgroup/cpu/system.slice/cpu.rt_runtime_us
Still blocked:
# docker run -it cyclictest bash root@231fbb116315: ~ # chrt -f 1 w chrt: failed to set pid 0's policy: Operation not permitted
3. Update cpu.rt_runtime_us for the new container:
# echo 900000 > `find /sys/fs/cgroup/cpu/system.slice|grep docker|grep scope|grep cpu.rt_runtime_us`
Now it works:
root@231fbb116315: ~ # chrt -f 1 w 11:01:56 up 26 min, 0 users, load average: 0.08, 0.05, 0.05 USER TTY LOGIN@ IDLE JCPU PCPU WHAT
Yes, it should be made easier…the question is at what level do we integrate this; Docker or orchestration.
For more info, see this Red Hat Bugzilla.
Getting Started with Performance Analysis of Docker
Docker introduces some intriguing usability, packaging and deployment patterns. These new patterns offer the potential to effect massive improvements to the enterprise application development and operations specialties. Containers also offer the promise of bare metal performance while offering some amount of isolation as well. But can they deliver on that promise ?
Since the early part of January, the Performance Engineering Group at Red Hat has run huge amounts of microbenchmarks, benchmarks and application workloads in Docker containers. The output of that effort has been a steady stream of lessons learned and advice/guidance given to our product architects and developers. How dense can we go ? How fast can it go ? Are these defaults "sane" ? What NOT to do…etc.
Disclaimer: as anyone who has worked with Docker knows, it's a project under heavy development. I mention that because this blog post includes code snippets and observations that are tied to specific experiments and Docker/kernel versions. YMMV, the answer of course is "it depends", and so on.
Performance tests we've pointed at Docker containers
We've done a whole bunch of R&D testing with bleeding edge, "niche" hardware and software to push and pull Docker containers in completely unnatural ways. Based on our choice of benchmarks, you can see that the initial approach was to calculate the precise overhead of containers as compared to bare metal (Red Hat's Project Atomic will support bare metal deployment of containers). Of course we are also gathering numbers with VMs to compare and containers in VMs (which might be the end-game, who knows…) via OpenStack etc.
Starting at the core, and working our way to the heaviest, pushing all the relevant subsystems to their limits:
- In-house timing syscall benchmarks (including vdso), libMicro
- Linpack, single and double precision, Streams
- Various incantations of sysbench (oltp and cpu)
- iozone, smallfile, spinning disk, ssd and NAND flash
- netperf on 10g and 40g, SR-IOV (pipework)
- OpenvSwitch with VXLAN offload-capable NICs
- Traditional "large" applications, i.e. business analytics
- Addressing single-host vertical scalability limits by fixing the Linux kernel and fiddling some bits in Docker.
- Using OpenvSwitch to get past the spanning-tree limitations of # of ports per bridged-interface.
All of these mine-sweeping experiments (lots more to come!) have allowed us to find and fix plenty of issues and document best-practices that we hope will lead to a great customer experience.
BTW if you're interested in serious, low level, Enterprise-grade performance analysis and tuning for Linux containers (or in general!), let's have a chat @DockerCon … I'll be one of the guys in a Project Atomic T-shirt 🙂
Unique Docker Philosophies
- Ease of use: Docker automates the use of existing Linux kernel technologies into an easily consumable format. Setup and administration of traditionally disjoint subsystems (cgroups, namespaces, iptables, selinux) are encapsulated by Docker.
- Packaging: Docker specifies an image/packaging format that allows an application to be packaged with it's full userspace requirements. No longer is there a necessary interaction between system-level packages (other than the kernel) with the containerized application. The application sees only what is provided inside the container. This can be for example, a specific version of gcc or php that differs from what the host OS provides. I keep drawing an analogy to BIND "views".
Performance interests aside, those are the 2 main selling points for me, and the benefits of those cannot be overstated.
Surprise, we added some enterprise-y stuff
Docker learns about systemd
Red Hat has taught Docker to use systemd, rather than sysvinit. I mention this because (depending on who you're talking to) it may be controversial. But I believe that the true promise of containers on Linux relies on specific capabilities that systemd provides: at least init dbus messaging, remote capabilities, cgroups API, remote journaling.
Docker systemd unit-file override:
- systemd supports ".d"-style overrides for installed unit-files. This is the correct way to customize the defaults for any systemd unit-file. Overrides go in /etc/systemd/system/.
- I need an override for my testing, because I want to use my own bridge device and I want to play with the MTU as well. By default, Docker creates a bridge called docker0 and assigns IP addresses from that pool, useful for development, not production. For production, I guess folks will want to set up their own bridge (or pass through a device, macvlan, whatever).
- Assuming you have a bridge that you want to use, create a new systemd unit override file called /etc/systemd/system/docker.service. Here is an example where I've set Docker to use a bridge named 'br1' and I also added '-D' to enable debug logging for the Docker daemon. br1 is on my test network, on an IP range that I control. Finally, I've bumped the MTU to 9000 for some throughput tests…
ExecStart=/usr/bin/docker -d --selinux-enabled -H fd:// -b br1 -D --mtu=9000
Also Stephen Tweedie spotted unnecessary memory consumption in systemd mount/umount handling, which was fixed in record time by Lennart Poettering 🙂
Docker learns about SELinux
Red Hat has brought SELinux support to Docker. If you've been using Red Hat products for any length of time, you know security is a first order concern for us. Look at the stats for critical CVE reponse time…adding SELinux support to Docker should come as no surprise 🙂 Shout out to the wizards in Red Hat's Security Response Team, btw.
After the initial bring-up, SELinux support has been fairly painless for us in the Performance Group. Dan Walsh is doing a talk called "SELinux and Docker" at DockerCon next week (June 10, 2pm, actually). To give you a sense of how serious Red Hat is about containers and Docker, I should also mention Red Hat's CTO Brian Stevens is doing one of the keynotes and we're Platinum sponsoring. Here's the very high level picture:
Dockerfile for Performance Analysis
What is a Dockerfile?
- Conceptually, a Dockerfile is like a kickstart file for Docker containers. It includes the precise recipe by which Docker builds your container.
- Link to Docker's Dockerfile Documentation
- Link to Red Hat's Resource Management and Linux Containers Guide
Why create a Dockerfile specifically for Performance Analysis?
- One of the core principals of Docker images is that they are absolutely as small as possible. This is because when a user wants to use your container image, they must pull it over the network. Docker hosts a registry at http://index.docker.io. Folks may stand up their own internal registries as well, where bandwidth is a bit less of a concern, images can contain site-specific customizations, intellectual property, licensed software, etc.
- Our engineers have been working hard to reduce the base image size. Therefore, the base images include the smallest usable package set, plus necessary tooling/package management utilities (yum) to pull in anything else the user needs inside their containers. Think @core on steriods.
- Because of the size constraints on the base image, we have to layer on our usual set of Performance Analysis tools via Dockerfile rather than kickstart.
- A very common question I get from the field is to provide a precise list of performance analysis packages/tools that I would recommend in their base RHEL images. So I put a slide in the Summit deck this year:
Example Dockerfile
It's not all that complicated, but includes lots of helpful utilities for characterizing workloads running inside containers. You might see that sysstat is missing; that's because I monitor that information on the host. This is one critical differentiation between virtualization, and containers: the VCPUs of a KVM guest exist as processes in the host. With containers, the actual containerized binary shows up in the process list of the host. Note: the PID namespace ensures isolation of process tables between containers.
FROM rhel7:latest MAINTAINER perf <email@example.com> RUN yum install -q -y bc blktrace btrfs-progs ethtool gcc git gnuplot hwloc iotop iproute iputils less mailx man-db netsniff-ng net-tools numactl numactl-devel openssh-clients openssh-server passwd perf procps-ng psmisc screen strace tcpdump vim-enhanced wget xauth which RUN git clone http://whatever/project.git ENV HOME /root ENV USER root WORKDIR /root EXPOSE 22
You might also notice that I'm installing numactl and hwloc. That's because recent versions of Docker provide access to sysfs hardware topology tables from the host, allowing you to apply similar tuning techniques as you would on bare metal on containerized processes. We had some pretty funny test automation explosions when sysfs hardware topology was not exposed 🙂 Side note, you can't tune IRQ affinity from a non-privileged container, but luckily IRQ balance really does a great job these days (even knows about PCI-locality). Privileged containers CAN program IRQ affinity.
CPU and memory affinity is another important differentiation between VMs and containers. In a container, core1 is core1 on the host, core2 is core2 etc (depending on your cgroups config). With VMs you apply specific vcpupin/numatune/emulatorpin commands in order to ensure VCPU threads and their memory utilize specific CPUs/memory banks. The process of properly applying affinity to KVM guests is well-documented in Red Hat's Virtualization Tuning and Optimization Guide. Naturally, when we characterize VMs and containers inside VMs, we often apply much of that.
How to build a container with the Performance Dockerfile
# time docker build --no-cache=true -t r7perf --rm=true - < Dockerfile_r7perf # docker run -it r7perf bash root@7d7b16277784: / # exit
How do I add my benchmark/tool/workload to this Docker container?
- Ideally, a pre-configured set of scripts would be committed to your own git repo, and pulled into this container automatically in the Dockerfile (RUN git clone http:///whatever/project.git). This is our approach.
- Add a RUN command to the Dockerfile that uses yum, wget, git or similar to pull in, install and configure your software.
- Run a container interactively, then pull down the benchmark manually. This is our fallback for some of the more challenging/complex benchmarks and under-load analysis.
How to get a benchmark running inside a Docker container
Let's take for example, sysbench.
- I've built RPMs for sysbench for RHEL6 and RHEL7 and committed them to our git repository. I've also committed my driver script called run-sysbench.sh. (this isn't mandatory, but using git makes things a LOT easier).
- You can add a RUN statement to the Dockerfile that wget's your benchmark/tarball from somewhere, or a RUN that does another git clone of some other repository.
- However you would normally transfer your code to a new machine, you can do the same thing in the Dockerfile.
- Once the container build is complete, launch a container, and kick off your workload. run-sysbench.sh could be any driver/wrapper script that you've got.
host# docker run -it --privileged r7perf bash
container# yum install -y bench/sysbench/rhel7/*rpm mariadb-server mariadb ; cd bench/sysbench
container# ./run-sysbench.sh oltp docker
...run-sysbench.sh completes and spits out an output/logfile that it copies off the container (rsync, ftp whatever).
- That's it. When the script finishes and you've copied off the results (part of run-sysbench.sh), you can 'exit' the container.
- Astute observers will have noticed that I snuck '–privileged' onto the command line above. That is because my run-sysbench.sh wants to drop_caches, and that's not something permitted to a container by default. As an alternative, instead of using privileges, a container could ssh into it's host machine as root and drop_cache from there. See Docker source capabilitiesdaemon/execdriver/lxc/init.go for the additional capabilities afforded to "privileged" containers.
- Fun example: create 100 containers running apache, in 14 seconds 🙂
# time for i in $(seq 100) ; do docker run -d r7perf /usr/sbin/httpd -DFOREGROUND ; done
real 0m14.159s
user 0m0.386s
sys 0m0.386s
It's not very often that a new technology comes up that creates a whole new column for performance characterization. But containers have done just that, and so it's been quite the undertaking. There are still many tests variations to run, but so far we're encouraged.
That said, I have to keep reminding myself that performance isn't always the first concern for everyone (*gasp*). The packaging, development and deployment workflow that breaks the ties between host userspace and container userspace has frankly been a breath of fresh air.


text: You know how some couples are just great together? Being around these two made me smile, they are so in love. One of my favorite parts about photographing couples is getting to hear all of their stories, how they fell in love, and then getting to watch their family grow. I've heard that congratulations are in order to the two of you, expecting a little one! I'm so excited for your family to become bigger, your love will only grow.

subset: commoncrawl


text:
Local Obituaries
Stanley Arkin, builder, former Miami Beach commissioner, dies at 82
When Stanley Arkin closed his construction business in 1997, he briefly considered retiring, focusing on his family and his hobbies including model trains.
But his retirement didn't last long.
Arkin — whose construction business built parts of Mount Sinai Medical Center, the Miami Beach Hilton and Miami Beach Jewish Community Center, to name a few — soon formed Arkin Consulting Inc., working on projects for what is now Jungle Island, Braman Management and the Adrienne Arsht Center for the Performing Arts.
Arkin, who also served as a Miami Beach commissioner from 1984 to 1991 and was a life member of the University of Miami's Board of Trustees, died Saturday night at age 82, just a week shy of his birthday. His son Robert Arkin said his father's health declined after his wife of 56 years, Jill Arkin, died in June 2014.
"He died of a broken heart," he said. "He had a really hard time after my mother died."
Added Stanley Arkin's older son Bradley: "When she died he totally collapsed."
Arkin was born in Manhattan on Aug. 28, 1932, and moved to Miami Beach with his parents and two older siblings in 1934. His father, Joseph Arkin, began Arkin Construction in 1935.
Stanley Arkin graduated from the University of Miami in 1953 with an accounting degree and joined his family's construction business shortly thereafter. He became president of the company in 1994.
Bradley Arkin said his dad loved Miami Beach and became involved in boards because "he always wanted to help make things better."
In 1984, Arkin was part of a group of Miami Beach activists who sought counsel from the city of Miami when the financially beleaguered Beach was poised for a spate of development similar to growth that Miami had experienced.
"The idea is new and kind of exciting," Arkin said at the time. "It's going to be hard to do. It's the first time in 49 years I've seen both sides sit down and talk."
That same year he was appointed to fill a seat left vacant by a commissioner who resigned to run for a Senate seat.
After being appointed, the self-proclaimed "apolitical man," told the Miami Herald that he has "looked at city government from the other side."
"I think that we have an opportunity to see great change," he said. "It will take me a while to learn my way around as a commissioner. But I am exhilarated.''
He served on the commission until 1991, when he resigned.
In the late 1990s, Stanley Arkin got involved in the Performing Arts Center Trust and played a key role in the construction of the Adrienne Arsht Center for the Performing Arts as a consultant.
Arkin's job was to oversee the bidding process for contractors. In 1997, he led negotiations to hire Church & Tower, owned by the family of Cuban National Foundation leader Jorge Mas Canosa, as a construction manager.
He managed to get an agreement for less than $4 million after talks started at $7.5 million;, according to news reports at the time. Arkin brought the contract in at less than $4 million.
Arkin also dedicated a lot of his time to the University of Miami and Bascom Palmer Eye Institute.
"Stanley Arkin was just a wonderful human being with a passion for Bascom Palmer and a life member of the Board of Trustees," said Donna Shalala, former University of Miami president. "He was always the first to volunteer."
Norman Braman, auto magnate and longtime friend of the Arkins, said "Bascom Palmer wouldn't be what it is today without Stanley."
While his son said he loved being involved in the community and helping others, he also loved being with his wife and family.
When his wife — whom he met at a Fontainebleau pool party in 1957 —was diagnosed with dementia, he became a full-time caregiver and was with her until the end.
"I don't think there was a more dedicated family man and husband," Braman said.
Added Bradley Arkin: "They were so affectionate and so in love."
In addition to his sons Robert, Bradley and Gregory Arkin, he is survived by three grandchildren and his brother Jules Arkin.
Services will be held at noon Thursday at Temple Beth Sholom, 4144 Chase Ave, Miami Beach.


text:
|Copic Colors Used
Thank you for stopping by. I love reading your comments!
Also Entered:
Penny's Paper-Crafty Challenge
Crafty Girlz Challenges
Let's follow the path toward colorful and playful art together, learning as we go.
|Copic Colors Used
Jean, Gorgeous, colored beautifully. Love the wonderful colors and the pretty flowers.
Hugs Pat
Thanks, Pat. I think this is a beautiful image to color!Delete
i am trying to find out where to et this digi - can you please help. thanks
Dennis, I have it linked above. Just click on the word "Fifi"!Delete
I'm behind as usual! I love that background, Jean, and such beautiful coloring of the image!ReplyDelete
Thanks, Janet. Me too LOL!Delete


text:
On July 5, 2016, ALJ David P. Shaw issued Order No. 3 in Certain Magnetic Data Storage Tapes and Cartridges Containing Same (Inv. No. 337-TA-1012).
By way of background, this investigation is based on a May 27, 2016 complaint filed by FujiFilm Corporation and FujiFilm Recording Media U.S.A., Inc. alleging violation of Section 337 by way of unlawful importation into the U.S., selling for importation, and/or selling within the U.S. after importation certain magnetic data storage tapes and cartridges containing the same that infringe one or more claims of U.S. Patent Nos. 6,641,891; 6,703,106; 6,703,101; 6,767,612; 8,236,434; and 7,355,805. See our May 31, 2016 and July 8, 2016 posts for more details on the complaint and Notice of Investigation, respectively.
According to Order No. 3, ALJ Shaw set October 2, 2017 as the target date for completing this investigation (which is approximately 15 months after institution of the investigation). ALJ Shaw further noted that the initial determination on violation will be due on June 2, 2017.


text:
Hello Friends,
The 'Fierce Wind' Limited Edition, hand-numbered old-fashioned CD is now available. Head on over to the music section and order it there, or download the digital version if you don't do CDs anymore.
In other news, I'm headed off to Ireland, Scotland, and England to do a few shows and some 'research' for the next album, so check back for some updates in a week or so.
Also, several of you have been asking about a CD release show. Yes, there will be one at the end of June, but I thought I'd give you all a little time to learn your sing-along parts before the big gig.
Talk soon.
LIMITED EDITION EP now shipping, UK shows!
Hello Friends,


text:
Kubilay Üner is a person of boundless optimism and enthusiasm, with razor sharp vision and overflowing passion. His new opus INN PARADISO shows us just how wildly inventive an electronic composer with no perceived boundaries can be!
"I do a lot of sketching. Always have... Over the past year or so a new theme emerged in my sketches - my ideas were getting thicker, denser, more complex, more layered, more distorted, perhaps even more angry. But also extremely vivid, strong, alive, vibrant. They were all about 'more is more', so I called it my 'maximalism'..." (Kubilay Üner)
"Maximalism" is a great way to describe this music. When your source material ranges from sampled escalators, wolf packs, cat fights, trains, and random street punks, to a cast of nearly 30 brilliant musicians scattered across the globe performing on strings, brass, winds, and other bizarre and exotic instruments, how could you call it anything else. What's really amazing about all this, though, is that INN PARADISO is anything but a mash of jumbled sound. It's a searingly distinct vision, tight and coherent, and though the composer claims "it's not about anything," it feels like a memory of an undiscovered sonic paradise!
"I realized I probably borrowed a lot from painters like Gerhard Richter...who starts with a layer of actual paint instead of, say, a pencil outline sketch of the shapes, then scrapes, paints again, scrapes again, scrapes some more. Similarly, I kept adding to the original layer, whatever it was, by recording over it, processing it, recording again, processing the whole thing, doing it again, layering again, and so on. All this is done based entirely on the question, What do I want to hear next, and where?" (Kubilay Üner)
By sculpting this music in such a way, Üner has revealed a kind of pan-experiential sonic masterpiece. The effect upon listening is almost like an unveiling of music that exists all around us in our daily environments. Indeed many of the sources come from familiar places, but we almost never stop and listen to them as music. INN PARADISO offers us the chance to hear the familiar as unusual, and the unusual as familiar.
"Since these are neither songs nor film scores, this music is just about creating a "music-scape" for you to wander around in. What you see, hear, make out, even if it's a real sound like a wolf or an escalator, is just there to sound the way it does. There's no story, no meaning, just a ballet of sound gestures coming together to create a dance. You make your own story - or just enjoy the movement and colors.
"One more thing: This is listening music - pour a drink or a cup of tea, put on a good pair of headphones or sit down in front of your stereo, and enjoy the trip!" (Kubilay Üner)
OK Kubi. I'm in!
—Daniel Rosenboom, founder of Orenda Records


text:
|Home | Davis Podiatry Center | Dr. Ferragamo, D.P.M. | Services | Specialized Treatments | Foot Disorders
Foot Disorders | Neuropathy
What is Neuropathy?
In diabetic peripheral neuropathy of the feet, usually you will have begun to notice sensory changes such as numbness or tingling in your toes. At first these symptoms will come and go, but then they will be constant. These unpleasant symptoms may interfere with your going to sleep, or cause you to awaken from your sleep. Over a period of time, these sensory disturbances may cause such a loss of sensibility that you will not feel how tight you shoes are or if you have a pebble in your shoe. Weakness of the foot muscles may also occur causing hammertoes or the arches of the feet to collapse.
Neuropathy is the leading cause of the ulcerations or holes that occur
in the feet. Neuropathy is the leading cause of infections in the feet.
Neuropathy is the leading cause of the loss of toes and, with advanced
cases, amputation.
Why are Nerves in the Diabetic Compressed?
The diabetic's nerves are susceptible to compression because the nerves in a diabetic are swollen due to excess sugar in the nerves that draw in excess water. If a nerve swells in a place that is already tight, like the areas described above, then the nerve becomes pinched, or compressed, causing symptoms.
What are the Symptoms of Nerve Compression?
In the foot, the posterior tibial nerve travels through a boney and fibrous tunnel on the inside of the ankle. This nerve supplies the entire bottom of the foot, including the heel. Compression of the posterior tibial nerve can result in numbness or tingling of the heel, the arch, the ball of the foot, and the bottom and tips of the toes. Continued compression of the nerve without treatment can further lead to muscle weakness, or atrophy, of the small muscles to the toes causing cock-up or hammertoes.
What is the Relationship between Neuropathy and Nerve Compression?
The symptoms of diabetic neuropathy, of the sensorimotor polyneuropathy type, the most common type that we have been discussing thus far, are numbness and tingling, and weakness, and are essentially the same as those of nerve compression.
But what if there are more than one nerve compressed in the ankle and foot at the same time? Knowing that diabetes makes nerves susceptible to nerve compression and knowing that there are many areas of tightness that occur normally in everyone, it is possible that the diabetic could have more than one nerve compressed in each foot. It this were to be true, then multiple sites of nerve compression along the path of the nerves would give a stocking pattern to the symptoms of numbness and tingling.
Another way to think about the relationship of neuropathy and nerve compression is that diabetes creates the neuropathy according to some metabolic process. This neuropathy, then, creates the circumstances that allow nerve compression to occur. It is well known and accepted that nerve compression can cause the symptoms of numbness, tingling and muscle wasting of the muscles of the foot. It is possible, then, to think that the nerve compressions are superimposed upon the underlying neuropathy. This means that at some point in time, both neuropathy and nerve compression may exist together, but the symptoms may be due to the sites of compression.
What Type of Surgery can be Done?
How Does this Type of Surgery Help the Nerve?
The surgery to decompress the nerve does not change the basic, underlying (metabolic) neuropathy that made the nerve susceptible to compression in the first place. When the surgical decompression is done early in the course of nerve compression, restoration of blood flow to the nerve will stop the numbness and tingling. When decompression is done later in the course of nerve compression, and nerve fibers have begun to die, decompression of the nerve will permit the diabetic nerve to regenerate.
Of course, if one waits too long to decompress the nerve, recovery may not be possible. If you already have ulcerations on your feet, or have lost toes, then very little sensation may be recovered because the damage to the nerve has become irreversible.
Who is a Candidate for this Type of Surgery?
If the patient is seen sufficiently early in the course of nerve compression, it may be possible to relieve some of the pressure upon the nerve by wearing shoe inserts (orthotics) for the feet. Special instruction is given to the patient in terms of daily inspection of the foot for early signs of skin breakdown or infection. When the Quantitative Sensory Testing demonstrates sufficient sensory loss, special shoes may be required to protect the feet. There are some medications that can be given to relieve the discomfort of the neuropathy on a temporary basis. And of course, you must be sure that your blood sugar level is the best that it can be.
If the sensory loss progresses to the point where you have numbness and tingling throughout the day or it even wakes you up at night, then you may be a candidate for surgical decompression of your nerve. The ideal candidate does not wait until there is no feeling left or until there is already an ulceration present. The ideal candidate seeks surgical consultation while there is still time to reverse the damage to the nerves.
Where Is The Surgery Performed?
How Long Does the Surgery Take?
From the time you enter the operating room until the time you enter the recovery room is about two hours. You will stay in the recovery room for another hour. These times will vary for individual patients.
Do I have To Be Put To Sleep?
Is the Surgery Painful?
A long-acting local anesthetic will be put into your incisions so that when you awaken not only will there be very little pain but you may not feel your foot at all.
When the local anesthetic wears off, in about four hours, you will begin to feel your foot. If this becomes painful for you, you will have been given pain medication, which you may take for after the surgery if needed. You may have some increased pain as you begin to walk again.
When the nerves that have been "asleep" awaken, you may experience hot or cold or shooting pains in your toes. This is a good sign as it shows recovery, but it may still be uncomfortable for you. Medication can help these feelings too.
Do I Have to be Hospitalized?
When Will I Have Quantitative Sensory Testing?
The American Diabetes Association strongly suggests that every diabetic has quantitative sensory testing every year and so you may already have had your testing done. It must be done before surgery. So, if you have not already had it, it can be done within two weeks of your office consultation. The testing takes about an hour.
You will have testing done after surgery, too. Usually, it is done at about six to twelve weeks after surgery to document that neural regeneration is occurring. This will also document that the operated foot is improving and help us determine if you should proceed to have surgery on the opposite foot.
What are the Risks of this Surgery?
With any surgery, there is always the risk of bleeding, infection, scar formation, the unpredictable nature of the healing process and failure of the procedure to achieve its desired goal.
Unique to the surgery you will have is the possibility of having a painful scar, of having apparent worsening of your symptoms as the diabetic nerve regenerates and delayed wound healing.
Overall, nerve decompression surgery is a valuable procedure in reversing
the complications of diabetic neuropathy a is a new source of optimism
is reversing potential complications of the feet in diabetics.
Foot disorders & how we can help
Helpful Resources
Appointments and
|Tracy Ferragamo, D.P.M • 2925 Spafford St., Ste. A, Davis, CA 95618 • 530-753-9080
|Copyright © 2007 DavisPodiatryCenter.com
Web design/development Comentum


text:
Thursday, April 23, 2015
5k: Ford Zetec Quad-Carb'd: 1963 Morris Minor
It would be easy to mistake the Minor for an Austin Mini imitator (family sedan built in large quantities, many variants available, old styling lasting for decades), but the Minor is much larger and used a front-engine rear drive setup instead of the Mini's front-front.
Powering this Minor is a 2.0 liter Ford ZX2 Zetec inline-4 fed via 4 carburetors and pushing out an unknown number of ponies into a 5-speed transmission from a Mazda MX-5. It should make for a fun times in the 1800 lb Minor.
The interior isn't exactly mint condition, and has some dangling wires under the dash that could be easily cleaned up. Additionally, it features an aftermarket steering wheel and a set of seats sourced elsewhere.
See a cooler way to cruise around and surprise everybody? firstname.lastname@example.com
4 comments:
- BobinottApril 23, 2015 at 7:25 AM
Now that looks like $4K worth of FUN! It took me a while to spot the gear lever. Does the MX-5 transmission shift from the end of the box?ReplyDelete
Despite some horrific personal decisions that make me want to hurl, I love me some Roman Polanski. Unfortunately, only the Jag that's in the film appears in the trailer.ReplyDelete
Man, if I had any space in my garage I'd be selling my scooter off and replacing it with this. At 4k, you can't loose unless its somehow worse than it looks in the pictures.ReplyDelete
Commenting Commandments:
I. Thou Shalt Not write anything your mother would not appreciate reading.
II. Thou Shalt Not post as anonymous unless you are posting from mobile and have technical issues. Use name/url when posting and pick something Urazmus B Jokin, Ben Dover. Sir Edmund Hillary Clint Eastwood...it don't matter. Just pick a nom de plume and stick with it.
III. Honor thy own links by using  description of your link
IV. Remember the formatting tricks italics and  bold
V. Thou Shalt Not commit spam.
VI. To embed images: use [image src="http://www.IMAGE_LINK.com" width="400px"/]. Limit images to no wider than 400 pixels in width. No more than one image per comment please.


text:
ProPhorce™ AC 628 is a highly concentrated liquid feed acidifier based on formic acid with strong acidifying and antibacterial properties. No ADR restriction due to the use of buffered organic acids.
This website uses cookies to improve user experience. By using our website you consent to all cookies in accordance with our privacy policy.


text:
BelleayreHome Menu
Belleayre Calendar of Events
Why Not Wednesday - $32 Lift Tickets
- Date: 02/13/2019
Every non-holiday Wednesday of the season is a "Why Not Wednesday" with Dasani! That means a one-day lift ticket is just $32 when you present our friendly ticket sellers with an unopened Dasani bottled water. Not Valid December 26, 2018 and February 20, 2019.
We only accept full, unopened products. Please do not bring us empty or partially emptied products, they will be refused. Thank you for your cooperation!
Your water contribution will be donated to food pantries, relief efforts, fundraisers, and not-for-profit groups for their efforts in improving our region. Thank you from Coca-Cola & Belleayre, for making someone's day brighter!


text:
The LinkedIn Killer: Wall Street Journal takes a STAB at itAugust 5th, 2009 | by Jason Alba |
How do you like that – I came up with a tre-clever title! Killer… stab… ah, I crack myself up :p
Seriously, let's talk about the notion of a LinkedIn Killer. On TechCrunch Michael Arrington writs about the WSJ Connect. Interesting article, which talks about the first attempt the WSJ had in this space. And now they are reviving their interest and doing it better and more seriously… you can read about it at Wall Street Journal Creating New "LinkedIn Killer" Called WSJ Connect. The comments (in general) are revealing and very good.
What do I think about ANY LinkedIn Killer? I think there are one of two things that could, might, maybe lead to a site/service that becomes the LinkedIn Killer.
First: Technology.
Having written the book on LinkedIn, I am asked regularly to check out someone's site or idea that is "better than LinkedIn." LinkedIn was developed a long time ago. Facebook came out around the same time but it had sexier, cooler, flashier technology. Facebook has led the social space with a lot of things, including F8, FB Connect, etc. It's just freaking cool (however, the fine print in their ToS's really negate all of the cool stuff they do, imho).
LinkedIn has been slow to adopt stuff. Heck, they took forever before they allowed you to put a Profile Picture up! They don't have a real open API (they have a few partners who they let in with their API, as opposed to FB's F8).
Their design hasn't changed a whole lot (I think), and the site might look a little dated compared to the newer sites and technologies out there.
However, I completely discount that any site is going to out-technology LinkedIn. They have built a critical mass and have the momentum that any wannabe-competitor is going to have a hard time trying to get close to, regardless of cooler technology.
TECHNOLOGY ALONE WILL NOT LEAD TO LINKEDIN's DEMISE.
(unless LinkedIn does not keep up with certain technologies, and their own technology gets too outdated – which many complain of but I don't think it's been critical to LI's site/health)
Second: Signups and Users.
This is the real issue, I think. This is the reason why no new startup is going to catch up with LinkedIn – I mentioned "critical mass" earlier. Another phrase that comes to mind is "first to market."
Think about the difference between Amazon.com and BN.com. Do you even know what/who BN.com is? Perhaps. Why the heck did Barnes and Noble sit around waiting for Amazon to prove themselves (and the Internet and ecommerce) before they did something about it. By the time Amazon became powerful, BN had a TON of catching up to do. And they are still doing the catchup. Look at this graph from Compete.com (Amazon is the blue line, BN.com is the green line):
There are a handful of sites who have the type of user base that could even seriously think about wanting to be compete in the same arena as LinkedIn. I *think* the WSJ is one of them – they have a very strong brand, they have readership, they are well-respected, etc.
The question is, can they actually get the people who go there to read the sage writing to actually do social networking stuff – even if it's highly professional social networking? Perhaps… but check out this comment from Amit Jain (Amit Jain on Twitter):
I just had a look at the new community launched by WSJ and I completely disagree that it can beat LinkedIn with its current design, flow & content. WSJ is trying to sell its products – news, blogs, videos, etc – more from it than enabling the business community interact. I tried to register on it and I was verified in 2 steps. Its too much work. Well, my point is that WSJ has still got to do much work to outbeat LinkedIn. The current version won't work.
If the mindset of WSJ's social networking play is the same as the WSJ monetazation play ("sell its products") then I think it's going to be an uphill battle.
I'd like to see this become a successful service, but I have a hard time believing that it will be a LinkedIn Killer … I should note that they are not the first big property that has tried to do this… but I bet you haven't heard of the others.
Two final notes:
- If the WSJ is serious I seriuosly recommend they engage Scott Allen as a consultant in this space. Perhaps even Shally (no-last-name-required) and Harry Joiner.
- Apparently they themselves called it the "LinkedIn Killer." Hm… that sounds interesting. Hyping up the team internally? They really drew a line in the sand with that one.


text:
Power to the People: How Humans are Heading Back to the Factory Floor
For decades, the trend in automotive manufacturing has been a dramatic increase in automation, in which complex robots replaced human operators to perform assembly and production tasks efficiently. But, today's new and emerging technologies are making it possible to bring human workers back to the factory floor.
In this Manufacturing Technology article, Light Guide Systems Founder and CEO Paul Ryznar outlines this trend and provides clear insight into how augmented reality tools can work with humans to tackle the ever-expanding range of options available in an assembly line.
When robots can't handle the high degree of individualization and variety available today, it's time to send the humans back in. Click here to read the full article.


text:
2015 Audi A6
2015 Audi A6: Imaginative engineering
The 2015 Audi A6 speaks of a highly imaginative engineering and most importantly Audi's premium case. It has set a few new principles in various fields of its specialization such as the lightweight body owing to the innumerable aluminum parts, the compelling and highly exceedingly proficient engines, the highly pumped case and the progressed sight and aid sound frameworks.
With it engine configuration similar to the other A6 editions, the 2015 Audi A6 1.8 TFSI has a weight of 3384.1 lb. (1,535 kg) excluding the pilot.
It has base cost in Germany with the 1.8 TFSI of about €38,400 whereas the A6 Avant 1.8 TFSI starts at about €40,900 and the A6 all road Quattro 3.0 TDI at just €56,650.
2015 Audi RS6 Avant


text:
16:20 Call from Police. A couple who had planned to walk from Dufton to Alston were lost in mist somewhere near Great Dun Fell. They had got themselves disorientated in bad conditions near the summit and lost the path, as the team were preparing to send a Land Rover to the summit and locate the couple, their family managed to track them down and bring them safely off the hill. 4 team members for 30 mins.
made with love from Appartamenti vacanza a Corralejo - Fuerteventura


text:
You don't have to spend a lot of money to get a decent phone. There are plenty of options out there for people who want to save a little cash but still need a phone that's fast, can connect to the internet, and looks nice, too.
Here's a look at some of the cheapest phones worth buying.
It's hard to find a phone cheaper than the Moto G. Motorola's second-generation model starts at $179, comes with a clean version of Android, and runs pretty smoothly, even though it's powered by components that are older than those in today's more expensive phones, according to reviews from CNET and Engadget.
In short, it's a nice, compact phone with easy-to-use software that won't cost you more than $200. You can also get the 4G model for $219.
The Moto E is even cheaper than the Moto G at $129 without any carrier subsidies. If you can get by with a basic phone and mostly care about using Android apps, the Moto E is a good choice.
In my review, I found its camera and lack of 4G to be the biggest weak points, but it's exactly what you would expect for a phone so cheap.
The Nexus 5 starts at $349, which is still much cheaper than most popular phones like the Galaxy S5 or iPhone 6, which cost anywhere between $500 and $800 without a carrier contract.
The best benefit from owning a Nexus phone is that you'll always get the most important Android updates before anyone else does. The Nexus 5 is a slim, attractive phone with a soft-touch back, a thin design, and a 1080p screen.
You can only use it on T-Mobile's network, but it's a great choice if you're on a tight budget.
You can now buy Amazon's Fire Phone for just $199.99 without a two-year contract, which is ridiculously cheap for a new phone. If you opt for a two-year contract with AT&T, you'll get the phone for just 99 cents — which is basically a giveaway.
The Fire Phone comes with a sharp screen, a 13-megapixel camera, a fast processor, and Amazon's own apps and services. Firefly, for example, gives you information about object around you by scanning it with the phone's camera. There are tons of cameras on the front of the phone which creates a 3D effect for some images. The phone hasn't been selling well, which is a large part of the reason it's so cheap, but it's an excellent value for the price.
The OnePlus One comes with many of the same components you'd find in most high-end Android phones, but it costs about half the price. The OnePlus One, which features a roomy 5.5-inch 1080p screen, a fast processor, and a comfortable yet attractive build only costs $300 off contract. It's pretty tough to find, but if you can snag an invite from someone you know who has the phone, it's totally worth your time.
The HTC Desire 816 is an attractive phone with a large, crisp screen. It also has boisterous front-facing speakers just like HTC's flagship One phone. You can grab a prepaid version of the phone for as low as $200 on Amazon.
If you're OK with keeping Apple's previous generation iPhone for two years, you can get it for free through AT&T and Verizon. Off contract, it still costs about $450, which is about $100 cheaper than newer flagship phones. The iPhone 5C comes in multiple colors and features a 4-inch Retina Display. It's not the greatest phone you can buy, but if you're tight on cash and love Apple's iOS ecosystem, it's a worthwhile choice.
You can get the iPhone 5S for $99.99 on a two-year contract. It's not Apple's newest phone, but it still comes with a gorgeous design, fast processor, excellent camera, and Touch ID fingerprint sensor. If you don't care about having the newest Apple product but want an iPhone, you should consider the iPhone 5S.
As is the case with most Lumia phones, one of the 830's selling points is its camera. It comes with a 10-megapixel camera with a Carl Zeiss lens, and only costs $99 on a two-year contract. AT&T also bundles a free Fitbit with the phone, which is a great value. Off-contract, the phone costs about $400.


text:
I enjoyed the format and interaction with other class members using online technology. Better than being there in person. The flow of the course was better than most in-person classes I have attended in the past.
– Keith W.
SharePoint training
Here are some of the key benefits of our SharePoint training:
- Attend the course from work or home
- Only requires Internet connection and web browser to participate
- Receive six months of access to course recordings and electronic lecture notes
- Professional microphone equipment used for audio broadcast
- Completion certificates available upon request
We offer all on-demand courses (asynchronous learning). Additionally, every course we offer is available for private delivery—either on-site, or online!
On-demand SharePoint training
Courses built for your schedule
We also offer on-demand education. With this approach, you take the course at a time of your choosing and at your own pace. These classes provide:
- Course screencasts, materials, code files and handouts hosted on a student only web site
- Completion certificates available upon request
- Send questions directly to your instructor
Additional questions for online or on-demand training
If you have additional questions about online or on-demand training, please check out our FAQ or contact us. We are happy to answer any questions you have and can alternatively set up a conference call to discuss your needs.
Private SharePoint training for your company
Why you should consider private training:
- Host a class at your office or attend online
- Include virtual employees
- Classes are "instructor led, student driven"
- All course sessions are recorded and are available to you for six months
- Fully customizable course syllabus
- Train small to large groups of people
- Target SharePoint training to various organization roles
- Fees start at $2,500/day
Contact us for a SharePoint training quote
Private training is a great solution for organizations that have several people who need to be trained, or who have needs outside of what our public classes offer. Building off our extensive consulting experience, SharePoint Experts can come into your organization and tailor training based on what you need. For example:
- The IT team needs an administration course, but wants additional time and attention spent on SharePoint services.
- Your internal graphic design team has the chops for branding SharePoint, but they need a couple of days with an expert to make sure they are on the right track and to get some questions answered.
- The site content owners are new to SharePoint and need to learn just enough about the product to successfully manage their content.
- You want to learn the tool, and get a jump-start on implementation using our guidance and support.
Customize your course by extending, shortening or combining classes. Also, provide your sites, requirements and designs (if they have been determined) for course demos. In the end, you will have code files ready for implementation for your site.
Private online SharePoint training
Training online is a great way to prepare people quickly and efficiently. Set your schedule based on employee commitments and time restraints. All of the benefits we offer to our public online classes are provided to your employees.
Scheduling a course
When you are ready to book a class, please send us a message for a private SharePoint training quote. We will work with you to schedule a private delivery onsite at your location or online.
Daily schedule
Once booked, the instructor will work with you to create a class schedule that best fits the needs of your organization and students.
Group discount for online courses
If you only need to train a handful of employees, attending one of our public online classes is usually the best option. In this case, group rates are available for our public deliveries of an online course.
Additional questions
If you have additional questions about private training, please check out our FAQ or contact us. We are happy to answer any questions you have and can alternatively set up a conference call to discuss your needs.


text:
|Lenore's Egg Salad
Have you seen the hydroponic lettuce in the stores? It's perfect for pulling off a few leaves at time for sandwiches, and since the roots are still attached, it holds for quite awhile in the fridge. Perfect for smaller households.
|Hydroponic Lettuce
Lenore's Egg Salad
6 eggs, hard boiled
6 T. mayo or salad dressing
1 stalk of celery, chopped fine
1 small shallot, chopped fine
6-7 Tony Packo's Sweet Hot Pickles & Peppers, chopped, plus a spoonful of juice from the jar
2 - 3 t. yellow mustard
Salt and Pepper
Old Bay Seasoning
Chop eggs, then add the rest of ingredients and stir to combine. Taste and adjust seasonings.


text:
0 Items
Close Recently added
You have no items in your shopping cart.
- Login
In order to better provide you with our services, we collect two types of information about our users: Personally Identifiable Information and Non-Personally Identifiable Information. Our primary goal in collecting information from you is to provide you with a smooth, efficient and customized experience while using our Site and services.
We will primarily use your Personally Identifiable Information to provide our services to you, including:
If you do not wish to receive e-mail, written correspondence, or telephone contact from our company, you may contact us via e-mail, or writing, or calling and request that you be dropping from these lists at any time. This request will not affect communications regarding orders placed with us or information regarding these orders.
If a third party has linked to our website without permission, we cannot be held responsible for any actions of that website or its contents.
All credit card information is secured using a standard encryption technology. The encrypted data is transferred over FTPS and is stored at a secured site over SSL. For further security on all accounts, each approved customer can be provided with their own User ID and password. Even though we use encryption and updated technology to secure all information, no data transmission can be guaranteed 100% secure over the Internet or any wireless network.
We do not knowingly collect, use or disclose Personally Identifiable Information about visitors that are under 18 years of age.
Bungalow, LLC reserves the right to modify or amend this Privacy Statement at any time without prior notice. We will post all changes to the Privacy Statement on our website. Modifications and amendments will apply to all past and current users and replace all prior policies. Bungalow, LLC's privacy policy is in effect on May 20, 2009.
Bungalow, LLC reserves the right to update or modify information at anytime without prior notice. By using this website, you will be subject to any terms and condition changes as they may occur. Bungalow, LLC retains the right to change prices and product packs when indicated. Due to a flux in order quantities, Bungalow, LLC cannot guarantee product availability at all times.
By using this website, you are acknowledging full responsibility for use of the site including all risks and costs associated with all necessary servicing or repairs of any equipment you might incur while using the scoutbags.com website. Bungalow, LLC will not be held liable for any damages related to your use of this site. By using the site, you also agree to hold Bungalow, LLC harmless from and against any claims, damages, costs, and expenses including attorneys' fees which may arise from your use of the site.
All scoutbags.com website contents including page headers, footers, images, graphics, and text are subject to trademark, service mark, trade dress, and any other intellectual property rights or licenses held by Bungalow, LLC. This website is intended only for use by commercial accounts for the purpose of resale purchases.


text:
- The ultimate expression of luxury is to choose the fabrics, pattern and details of the clothes you are going to wear. And we already know who you can turn to to get it.
- It is time for you to know the renewed service of Personal Tailoring by Massimo Dutti, who has given a 180 degree turn to the classic concept of tailoring.
- Since its launch in 2004, the signature dress customization service has met the needs of thousands of men around the world. Here's what it's all about.
A few years ago, before becoming a successful film director, Tom Ford decided to open channel to Oprah Winfrey, the most famous American television presenter. By then the Texan designer had already got everything under the creative direction of a handful of fashion houses and suddenly found himself without a job. In prime time he stripped her heart, realized her vulnerability and explained what her prescription had been against depression in her darkest days. In an almost perfect ritual, this elegant superhero explained what he was doing when hundreds of imaginary cataclysms engulfing his head brushed his shoes, a tie is knotted and wore a suit already is.. As if it were a magic formula, this game was able to comfort him: striving to look like a good man automatically made him one of them.
Like Tom Ford, in GQ we also know the immense power that keeps the garment star in our wardrobe, especially if it is made to measure. In the long century of history of our head we have searched for land, sea and air the perfect jigger, the most favorable cut, the secret of the most successful dimple and thousands of small details that make male tailoring in pure arithmetic . a plus B are equal to C, or what is the same, a good fabric and cut to length is equal to a man who dresses by foot Today we are ready to announce that we have finally found the firm embodies these mathematical values to which we aspire: the Personal Tailoring service Massimo Dutti.
Since its launch in 2004, the Spanish men's dress customization service has met the needs and tastes of thousands of men around the world. As experience is a grade, last year they decided to go a step further and enhance the service with interesting news. For example, the creation of small ornamental elements, from pieces of jewelry to handbags, handkerchiefs and watches, all indisputable symbols of elegance and chivalry in the code of good masculine dress. But there is much more.
If until now the option to customize a suit Massimo Dutti focused on more general aspects, now you can modify details as specific as how trouser or jacket, the interlining of American or choice of fabrics and linings. The Possibilities multiply and the result are unpublished costumes of pure, almost architectural lines; exquisite cuts and handcrafted finishes born by technical characteristics of hate couture as the 'half' or 'full canvas'.
For the relaunch of this line, the firm has worked hand fabric suppliers internationally recognized as Vitale Barberi Canonigo, Ashton Shirtings or Scabal. The three brands have worked hand in hand with Massimo Dutti's creative teams to perfect and develop the proposals of a new era in the tailoring of men. The final selection was chosen for each of the lines of the collection, seeking the delicacy and effectiveness for each of the pieces.
He forgets the most anodyne tailoring because it's time to claim the design that responds to a v-shaped pattern at the top and respects the proportions on both the shoulders and the sleeves and the low. Personal service Tailoring offers three complementary collections according to the needs we seek: Extreme Lux(extreme elegance as tuxedos and three – piece suits), Business Lux(overcoming the conventions through sartorial choices for the professional environment) and Country Lux (tailoring a 'twist' to weekend sports by camel coats or sweaters 'cashmere').
Suits, trousers, shirts, collars, cuffs, accessories … Whatever your claim, seek a garment for a special occasion or take a well deserved pampering, personalservice Tailoring Massimo Dutti has the answer. Some twenty stores in Spain and almost thirty in the world offer the ability to customize your tailored suit and look like a gentleman of the century. Also, you can already access the 'app' Personal Tailoring from your 'smartphone' and compose the clothes to your liking: the user must first create their own 'fitting', or import it from previous purchases, and it navigate between tissues, pockets , Buttons, initials, contrasts and thread of buttonholes. And all this from your home or from Oprah Winfrey's couch. All to keep your happiness through excellence and affordability.


text:
An attractive table makes any occasion feel more special. Layering your tablecloths is one way to make the table stand out. Use a solid color underneath and a light, lacy cloth on top, allowing the color to peek through the lace. For Valentine's Day, the bottom cloth might be red; for Christmas, make it red, gold or green. If lace is too formal for your occasion, try a patterned cloth layered over a solid.
Layering With Lace
Choose a lace tablecloth with a loose pattern that will let the cloth underneath show through.
Pick a solid tablecloth that is the same shape and slightly smaller than the lace cloth so that it will not be visible except through the lace.
Spread the solid tablecloth over your table; then put the lace cloth on top.
Adjust the cloths as needed to keep the solid cloth out of view at the bottom edge.
Layering With Pattern
Choose a solid-color tablecloth to anchor your design.
Pick a patterned cloth that is at least 6 inches smaller on all sides than the solid tablecloth.
Spread out the larger tablecloth; then center the patterned cloth on top. If the cloths have corners, match them up for a neater look.
Experiment with various colors and patterns in your layering. Perhaps you'd rather have a patterned cloth on the bottom and a solid one on top. Be brave and combine a stripe with a plaid. Use lace over a floral. The possibilities are nearly endless.


text:
Borderline Products
Regulatory Context
Most human medicines are clearly identifiable as such and are subject to European Union (EU) marketing authorisation procedures. However, there are some products where it is not so easy to distinguish a medicine from, for example, cosmetics, medical devices, biocides, food supplements, herbal products. These are known as borderline products.
Current issues in this Sector
Difficult to Determine Status
The person or company marketing a product has a responsibility to do so in accordance with the law. However, the status of many products on the "borderline" between medicinal products and cosmetics, medical devices, biocides, food supplements or herbal products can be very difficult to determine and in some cases, has led to requests for clarification of applicable EU law about the borderline to the European Court of Justice.
Product Claims
Information on the Internet about a borderline product and its uses is not excluded from the definition of the term 'advertising'. Many companies bring borderline products within the definition of a medicinal product by making direct or implied claims on the internet which cause the product to fall within the definition of a medicinal product.
Difficulties to be avoided
Incorrect classification of a product by a company can result in non-compliance with the applicable EU legislation. Risk of incorrect classification of a product can be minimised by a thorough review of ingredients, labelling, permitted claims (including those on the internet and product brochures), intended use, primary mode of action, similar licensed or registered products on the market and registration requirements of the product before launch onto the EU market. Advice may also be sought from the classification committee of the relevant competent authority.
Differences Between Member States
Within the EU it is difficult to get a clear understanding of the interpretation of the legislation relating to borderline products in each Member State, with the result that a food supplement product which is compliant in one country may not necessarily be considered a food supplement or compliant with the legislation of another country. Therefore, it is important that local requirements are identified in advance by personnel familiar with the localised interpretation of the legislation.
The most efficient route for your regulatory approval
Real Regulatory has up-to-date knowledge in key areas relating to borderline legislation and can assist you in dealing with current issues in product classification and in anticipating costly delays your route to market might encounter. Whether you need help with filing submissions, regulatory strategy, compliance issues, regulatory intelligence or commercialisation our highly experienced team of consultants can ensure you follow the most efficient route to regulatory approval.


text:
Alice In Wonderland star Helena Bonham Carter, who plays the wicked Red Queen in Tim Burton's 3D movie blockbuster, arrived on the fantastical green carpet in a mish-mash of a dress.ALICE IN WONDERLAND FASHION: THE BEST OF KITSCH CHIC
The famously-kooky actress opted for a Vivienne Westwood ensemble for last night's premiere, which oddly resembled three outfits sewn into one. In a white prom dress with billowing taffeta skirting, the Harry Potter star added a flesh-coloured waist-bow and then...oddly...a black-and-cream Beetlejuice-style shrug. With her brunette locks piled up on her head (we're guessing the windswept look was on purpose?), the 43-year-old added about half a dozen charm necklaces and pearls to, er, complete the effect. Helena, who sort-of lives with director Tim Burton (although in separate, but adjoining houses), was carrying a very cute clutch depicting her villainous character in the film - made especially for her.
Her kooky and oddball approach to fashion aside, she's a massive hit in the movie - and had us in stitches from start to finish. Other celebrities hitting the red carpet last night seemed to use the occasion to break every fashion rule in the book, with model Daisy Lowe (who we usually think always looks fabulous) arriving in purple underwear with a netted dress over the top. And singer Paloma Faith clearly was confused by 'royal premiere,' appearing to think it meant 'dress like a royal' as she arrived in a red velvet coat Queen Victoria would have been proud of.


text:
Operational excellence is a vital component of our success and part of the Ventas Advantage of superior properties, platforms and people. We've built strong partnerships with the industry's top care providers, researchers and developers by bringing them deep industry knowledge and reliable access to capital. The common thread among our partners is that they have created platforms for growth matched with relevant scale and best-in-class practices.
We invest in our partners and work collaboratively with them to achieve growth. By working toward common goals and supporting their success, we help them reach their business objectives.


text:
Lizzy Yarnold was left toasting a job a well done after the Olympic champion kickstarted her World Cup defence in style in Lake Placid, USA.
Yarnold, who has been nominated for BBC Sports Personality of the Year for her Sochi heroics, was making her competitive return since starring in Sochi.
And the 26-year-old didn't disappoint, posting the two fastest times as she recorded 1:51.13 minutes, 0.77 seconds ahead of Canadian Elisabeth Vathje who impressed on her World Cup debut, while the podium was completed by third-placed Janine Flock of Austria.
It was the ideal start for Yarnold who will once again be looking to finish top of the pile this World Cup season, however she admitted to feeling the pressure in the build up to her opening gambit.
"I love being here in Lake Placid but with it being the start of the season I was a bit apprehensive that I wouldn't be quick enough - that I wouldn't be able to think quick enough," she said.
"These younger athletes are so fast, so I knew I had to bring my best performance today, which I did."
Elsewhere in the women's event, Rose McGrandle, who qualified for the World Cup with three podium finishes in three Intercontinental Cup races in November, was also in fine form as she finished sixth with a time of 1:52.50 minutes.
And Laura Deas finished in 12th place in the first World Cup race of her career, the 25-year old showing the quality of her push in both of her runs as she was just 0.06 seconds off the start record in her second heat .
In a men's event that was dominated by the Latvian brothers, Martins & Tomass Dukurs, Dominic Parsons, who finished in the top-10 at his first Olympic Games in Sochi earlier this year, was 14th in an overall time of 1:49.84.
While Martins and Tomass Dukurs took gold and silver respectively, American Matthew Antoine claiming bronze, there was plenty more for Great Britain to celebreate as Ed Smith, who also qualified via the Intercontinental Cup, placed 18th.
© Sportsbeat 2014


text:
A food parcel can never go astray because we all need to eat no matter how old we are.
Grocery Gift Basket
- You can buy a woven basket or fold-up shopping basket and collect several different types groceries that they would not normally buy for themselves because of the cost.
- Fill it with all sorts of Christmas food items like nuts, short bread biscuits, nibbles, fruit mince pies etc.
- Fill it with tinned Christmas foods like tinned ham, oysters, Christmas pudding or Christmas cake
- Fill it with a variety of Sweets, chocolates, Christmas assortments of sweets.
- This does not have to be alcoholic, for the non drinker you could by an assortment of the non alcoholic drinks.
- Select an assortment of the small bottles of alcohol
- You could buy things for their caravan or traveling type of vehicle
- Luggage
- Tickets for a trip or weekend away
What NOT to BUY
Never ever buy someone a pet for Christmas unless you know for sure that they really want the type of pet you are buying. Too many pets are bought and left unwanted to roam the streets and end up being euthanized because of irresponsible people that do not understand the person they are buying for.


text:
Harry Benz Courroie LSD Light Dose
Make an Enquiry
Click to zoom
Lucy in the Sky with Diamonds.
Ah, the memories!... Sweet, sweet memories... So vivid, I can see barely clad women dancing rejoicingly in the sun and hear "Sgt. Pepper's Lonely Hearts Club Band" playing in the background, marching along the yellow brick road towards infinity and well beyond.
And those colours... Those beautiful colours... Not just yellow, but all of them! So magnificently glorious in all their psychedeliciousness!
We are different. Let's embrace it and have some fun with it. Even if you only shoot in black and white, we all need to let some colour back into our lives. In a unique and one-of-a-kind fashion, of course. After all, we are distinct individuals going about what we love in an altered state of awareness kinda way; which is what rangefinder photography is all about.
Données techniques


text:
Tue, June 26, 2018 7:30 P.M.
Parkway Central Library
Cost: FREE
Please join Randi on the launch date of her latest book, Aroused, which takes us on a guided tour through the intellectually arousing history and science of hormones. Randi's prior book Get Me Out: A History of Childbirth from the Garden of Eden to the Sperm Bank won high praise including being called by NPR a "delightful—and sometimes disturbing" romp through the history, fads, and science behind making and having babies. Randi earned her MD from Yale University, and MS and an MPH from Columbia University and an undergraduate degree, focusing in the history and sociology of science, from the University of Pennsylvania. Her articles have been published in the New York Times, among other national publications. She lives in New York City.
Parkway Central Library
1901 Vine Street (between 19th and 20th Streets on the Parkway)
Philadelphia, PA 19103