text:
James Reinders: Today I have Steve Teixeira, Product Unit Manager for Parallel Developer Tools at Microsoft, and we thought we'd talk a little bit about parallelism. Steve, we're both very enthusiastic about parallelism and encouraging developers to take advantage of the concurrency in multicore. What thoughts do you have now that we have Visual Studio 2010 out there for developers to use?
Steve Teixeira: I'm really excited about it. We've been working on this for a long time. We had a team squirreled away, led by David Callahan as kind of the architect, trying to figure out this parallelism thing and trying to figure out how we can get this shipping as a mainstream part of Visual Studio.
I'll be honest, even a couple of years ago, we had people that were thinking, Yeah, it's kind of this propeller head, tinfoil hat type of science fiction thing that they're working on. But over time as we would get out and get in front of developers and talk to them and explain to them the hardware reality in terms of sequential performance kind of near plateauing, got to go parallel to kind of get the most performance out of your software that message was really resonating with developers who were really concerned about it and we started to see pull from customers. Then, whatever kind of non believers we had internally started looking to the customers and going, Wow, this is something that people are really excited about in the next version of Visual Studio. So it's been a long time and it's the first version of Visual Studio that other than say threads or thread pools or some really core screened version of parallel computing the first version of Visual Studio that really has all this stuff baked in as a first class citizen.
Reinders: I think the things that are baked in, the things that we can present with Visual Studio and with Parallel Studio, I think they look pretty simple and some people might wonder what took so long. But you and I both know working with the teams how much complexity is underneath where you want to have that complexity so that it really works but you want to present an easy interface. One of those complexities that I'm excited about, we've talked about, is concurrency run time. Can you tell us a little bit about Concurrency Runtime and how you think about it?
Teixeira: If we would have just shipped the Concurrency Runtime, we probably could have shipped a year ago. But it's building all that stuff that makes it easy on top of the hard stuff that kind of makes it take a while to surface a really reasonable programming model.
So the programming models -- the Concurrency Runtime, first of all -- is a runtime that includes a couple pieces. One is a piece that sort of abstracts away the details of the underlying hardware so developers don't necessarily have to be concerned about the bare metal. And then there is a scheduler -- or any number of schedulers -- that can live on top of that, and those two components together are called the Concurrency Runtime. Now on top of that we can build a variety of programming models so, of course, at Microsoft we did PPL and we did asynchronous agents. At Intel, Intel is building TBB, perhaps OpenMP on top of the Concurrency Runtime, and we can invite other companies that have their favorite languages to also build on top of the Concurrency Runtime.
But the core idea is this notion of a task abstraction. So the idea is to get developers out of thinking the business of threads. I like to think of it this way: Threads force the developer to think like a computer. You have to think in terms of execution flow on the hardware when you're thinking about threads. This is how my program is going to run through execution on this particular piece of code. What tasks allow you to do is tasks allow you to think of bite size chunks of work in your program so you can literally divide your program up in bite size chunks of work and schedule those bite size chunks as tasks.
Reinders: The more the better.
Teixeira: Exactly, the more the better. We want to lower the cost for these costs because right now threads are really expensive to create and you don't want to just spin up any number of threads. You kind of have to be pretty smart about it: I've got this many procs. The system is this busy so I want to spin up this many threads.
The idea with tasks is, within reason, you should just be able to create these things like candy and spin them up and use them and then they go away. We're thinking it's close to an order of magnitude less expensive to create a task versus creating a thread.
Reinders: On the order of magnitude?
Teixeira: Only about an order of magnitude. I haven't seen the latest timings but I think it's something on the order of about -- you need a chunk of work on the order of maybe 10,000 cycles to amortize the cost of creating a task across all of that work. But that's actually a fairly small chunk of work in computer science speed.
So as long as you can divide up your work in that way -- and you can do that manually or you can do that through something like a Parallel_for loop and just say: Take this For loop and spin it off and create a bunch of tasks behind the scenes. The Concurrency Runtime underneath the hood will do a whole bunch of magic in terms of scheduling to make sure that those tasks are executed in the most efficient possible way on the hardware. But as a developer it's nice to know that that's happening for me but I don't really have to care exactly what's happening, except that it's going fast and it's being efficient.
Reinders: Two of the messages I hear you say that I hope are getting understood by every developer, I think, really important is this idea of programming in task instead of threads, a thread being: I've got a quad core. Let's divide up into four pieces. That's just not the right thing for an application we're trying to do. So I like to say if an application is asking how many cores there are, there is something wrong. It should just be creating lots of tasks. But Concurrency Runtime I like to think about what is the world like in the future if we don't have Concurrency Runtime? What problem is it solving? What disaster are we avoiding? It's something we often call just over subscription, right?
Teixeira: That's right. To give you some insight into how my twisted mind works, whenever this topic comes up, I think about that Star Trek episode of Kirk versus the Gorn, that lizard man, where they're fighting.
Reinders: He makes saltpeter.
Teixeira: That's right. Thank you. So Kirk and the Gorn are fighting and that's my mental image for what happens if you don't have something like the Concurrency Runtime is you've got all these applications on the system. All these applications are asking: How many cores are on the system? And let me just assume that I own all of them and then you end up with this nasty oversubscription problem where you've got an amount of work that you're trying to do on the system that's disproportionate to the amount of execution resources available on the system.
So there are two problems there: One is, don't do that. That's really bad. I think a task-based programming model helps quite a bit but it doesn't really alleviate the responsibility of developers and even people running software to understand like, I've got a bunch of stuff still happening on my system. Tasks help but they don't necessarily get rid of the problem of oversubscription. You could still have that kind of problem. It's just you'll probably spend less effort, as a part of this oversubscription like spinning up and shutting down threads or this sort of thing.
Reinders: I know some developers will use Concurrency Runtime directly but I think the most important use is for Intel's OpenMP, Microsoft's TPL, PPL, Intel threading building blocks. Then we've got things built on top of these that people don't think about like our math kernel library uses threading building blocks which now will use Concurrency Runtime. But before that someone makes a math library call and they happen to be using TBB or using Microsoft's OpenMP. Every one of these is creating a thread pool or thinking about it and it's not very hard to write a program and use a couple of models. Okay, if you're in a dual or a quad core, you're only got a thread pool of two or four. But we're going to see machines -- 8, 16, 32 -- very quickly here and just casually calling into three or four models, each one of them creates a thread pool of 32. There is going to be oversubscription really fast that makes no sense to do that. Concurrency Runtime solves that problem by being an agent that every one of these models calls to and says: I want some threads. I want a thread pool and then the thread pool can be shared and coordinated and we don't end up with ten different thread pools of 32, hopefully.
Teixeira: Yeah. The point you bring up, James , is a really good one. It's not just about separate applications running, spinning up their own thread pools. Even within one application, if I'm using libraries A, B and C and they're all parallelized they could all be parallelized by having their own little thread pools and we actually see this. We talk to customers and they're using some third party library. They don't necessarily have the source code for but you bring this stuff up in the profiler and you can see you launch this app and suddenly 80 threads are going wild.
Reinders: What is this library that I was using? I didn't know it created all those threads.
Teixeira: Totally. So the idea of having a base layer that's really going to manage the concurrency across the whole application that everybody can just understand that is there, that's available.
One of the things we did to kind of create that assurance for developers is we put the Concurrency Runtime in the Microsoft C runtime. So it's kind of if you're building an application on Windows, whether you're just strictly using the Visual C++ library or you're also mixing it or the Visual C++ compiler or you're also mixing in the Intel compiler. Either way you can link with the Concurrency Runtime. It's there, it's available. You don't have to do any weird like download this, get this. If you're building a C++ app it's just kind of there for you.
Reinders: So on top of Concurrency Runtime, then we see models like PPL, TPL, Intel Threading Building Blocks (TBB) coming in on top, all of them espousing task and looking that we use a task stealing algorithms underneath. How important is task stealing? Or how should I be thinking about task stealing?
Teixeira: Task stealing is a really cool concept. One of the issues you run into -- Let's take a simple case like a Parallel_for loop. So let's say I've got this For loop that iterates N times and I've got a quad core machine so kind of a naïve way to parallelize that. We check for the number of processors. We say we've got four. Let's take this For loop into and divide it by four chunks and we'll give each thread one-quarter of the workload and they will all run and that's a great way. In fact many applications still today parallelize that way.
Reinders: I call that 'static scheduling'.
Teixeira: Static schedule processors.
Reinders: Everybody gets a quarter. Hope it takes the same amount of time.
Teixeira: Then here is the canonical problem that blows up a static scheduling algorithm: calculating prime numbers. It's that kind of workload where you're close to zero. It's very little work to see if that number is prime and then when you get closer to infinity it becomes really quite a lot of work to figure out whether or not that number is prime.
So a static scheduling algorithm will end up with an imbalanced workload. You'll have a quarter of their workload will finish really soon; then the next quarter will finish a little bit longer, a little bit longer and a little bit longer. The problem is the time it takes you to solve that problem -- so you time to solution -- is as long as that longest piece of work. What tasks and work stealing allow you to do is break that task up into even finer grain pieces of work. So not just four; maybe it's going to break it up into 8 or 16 or 32 chunks of work and within that when one of those four things runs out of work it can steal some tasks from a neighbor. So let me draw you a little picture and kind of show you how that works. Let's say I've got my application and my application has some task or some thread that's running and executing. We'll call that a little button -- that looks like a little power button -- and I've got this global queue here and this global queue is going to store a bunch of tasks that I create so my thread can spin up Task 1, Task 2, Task 3, Task 4.
Reinders: One of these is running and the rest are sitting in the queue waiting.
Teixeira: In this case, these are all right now sitting in queue so I've got a Parallel_for and it's spun up all these things. What the underlying runtime will do is underneath the sheets it will actually create a thread pool, although you don't really have to know that, as a developer, but that's kind of what's happening, and the thread pool will figure out the right number of threads to create in order to handle the workload that is present in the queue.
So let's say I actually need four of these things so I'm going to create four buttons so each of these are threads that are running. We'll call this Worker 1, Worker 2, Worker 3 and Worker 4. So Worker 1 says, I see the global queue has an item. I'm going to go grab that and I'm going to start executing on it. Each of these ends up actually with their own local queues, as well. I'll tell you why in just a second.
So they end up with their own local queues. Let's say Worker 1 pulls off Task 1, starts executing it but Task 1 now starts spinning off its own task. Let's say it's a nested Parallel_for loop. So those will actually end up in a local queue, a queue that's local to Worker 1. So we'll call these 5, 6 and 7. We'll throw 8 in for good measure. So these end up on a local queue.
One of the advantages of this is this thing is managed by a kind of a rocket science, lock free algorithm for the global queue. You kind of what as little contention on that as possible so one of the ways we can avoid that is saying, Okay, if I spin up some work locally I'm just going to put this on my local queue and then when I'm done executing T1 I can just pop T5 off and then T6 and then T7 and T8, in that order.
One of the advantages of working in this direction is that you can some cash affinity so the logic is: If I spun up these tasks one after the other, they're probably kind of located near each other, either from a code standpoint or even from a memory consumption standpoint, so I'm going to get great cash behavior. Now, obviously these other workers are also going to be looking to the global queue so in this model Worker 2 might pop off Task 2 and Worker 3 might pop off Task 3, etc.
One of the cool advantages that work stealing give you is let's say this is my prime number calculation and let's say Worker 1 is running the far end of that prime number, the thing that takes a long time. So when Worker 2 through 4 finish, their task queues are empty. Let's say the global queue is empty. What Worker 2 will do is it will look at its neighbor and go, Hey, neighbor, you got any work that I can take off your hands. And it will see, yeah, I've got all this work here. It will actually start stealing from the top of the task so T8 will disappear from Worker 1's local queue and reappear here in Worker 2's local queue.
You notice how it pops it from the top. So in this case thinking, I don't want to mess with this good cash mojo that Worker 1 might have rolling so I'm going to steal some stuff from the top thinking that I'm going to have the least protuberance on my neighbor.
Reinders: Caches matter a lot.
Teixeira: Exactly. So in this way, you can image of all of these things pooling from the global queue, working off their own queues, taking from their neighbors when their neighbors have work and they're empty. What you end up with is in that prime number example I ended up with instead of having a very staggered finish and a very imbalanced workload you end up with all four things finishing at around the same time and your time to solution ends up being much more quickly.
Reinders: You've stressed the imbalance that can be in the equations. I do that when I describe this as well and then inevitably I'll get people who tell me, No, my problem is not imbalanced. But if Worker 3 is running on a core and one of our favorite things like a virus checker the same thing happens, the same imbalance. That worker thread won't be getting work done that's essentially global, even though we've spread it around for very good reasons and the stealing algorithms work just as well if the interference is coming from outside your application.
Teixeira: That's a great point.
Reinders: That turns out to be important because if you ignore that the amount of time necessary to run our application, at least this phase, is equal to whoever does the work the slowest, even if it's because a virus checker kicked in on core number three or something.
Teixeira: That's right. Or even internal the application you could end up in a situation where one worker starts false sharing for some reason and so you end up with the work just taking a lot longer between two particular workers. So you can end up with that situation, even within one application and even when you have calculations that ostensibly should take about the same amount of time. They may or may not.
Reinders: The task stealing algorithms come out of research in MIT -- Cilk research -- and that's, I think, become the way to build this sort of scheduler and such. But this is complexity that I know you and I enjoy talking about but it's under the hood. We've given them abstractions on we've got Intel TBB and we've got the Parallel Patterns Library (PPL) in Visual Studio 2010 that give this to C and C++ programmers. But Microsoft also took it a step further and has this task stealing available on .NET, the TPL.
Teixeira: The TPL. That's right.
Reinders: What is unique about TPL? What should we know about TPL? It has the same thing under the hood. Is there anything different?
Teixeira: Yes. Just to rewind a tiny bit. One of the points you bring up is a good one which is there are a lot of different entry points for this technology. Some people may want a program directly to the Concurrency Runtime and that's totally cool. Some people may want to use the parallel pattern library. Some people may want to use TBB. Some people may want to continue to use Open MP which people have used successfully for many years.
We have a couple of other programming models in native code. We also have asynchronous agents so this idea of having agents that are from a code standpoint disconnected with one another. They don't share any state and so you get around all these locking problems by having them not share state and communicate only via messaging. But at its core that is leveraging the functionality and the Concurrency Runtime as well.
Reinders: Is that Axum? Is that the name for it?
Teixeira: Axum is actually a variation on that theme which focuses on managed code. The asynchronous agents are shipping in the native code library in this release. So, yeah, we have some incubations and kind of technology that we've made available to the public via our dev labs download site.
Reinders: So we're getting this basic infrastructure -- Concurrency Runtime, task stealing -- and then we can have a proliferation of models on top and we can mix and match them.
Teixeira: That's the idea.
Reinders: That's really the great thing about having the infrastructure done right.
Teixeira: We've talked a lot about the native code stuff. You brought up Task Parallel Library (TPL) which is on the managed code side so within .NET TPL is very much like PPL. It's a task-based programming model. You can sort of empirically create tasks by saying, Task, create. Here is your lambda function to go execute. Or you can more declaratively state that task should be created by doing a Parallel_for or Parallel_for e-generation and then underneath the hood tasks start getting spun up, although from a developer standpoint all you said was Parallel_for.
Also, on the managed code side, we've got another interesting technology which is 'parallel linq', plinq. So linq is this interesting programming model that kind of takes some SQL-like elements and brings them into .NET languages so you can start doing SELECT statements and order by's and various SQL kind of things that feel very native to SQL people but they appear as a native looking part of the managed code programming models.
What we did in this release is take that even a little step further and you can add 'dot as parallel' to any of those queries and if it's on a local data -- in other words, it's not a query going off to a database, databases are really good at parallelizing already -- but if you're working with some local data, it will help you manage that data and it will use the available cores on your machine to basically lower your time to solution. It's almost like magic. It's like I had this linq query, I added 'as parallel' and suddenly it's like 3.9 times faster on my four-way machine.
Reinders: I know. In your demos, you go in and you just open up the code and you add this little 'as parallel'. It looks so simple and then you run it and it runs faster. But you've been seeing people get very creative with the uses of this. You were telling me that you've seeing things you weren't expecting because it's a pretty general mechanism.
Teixeira: That's right. It works on any IEnumerable dataset and so we're starting to see people go, Wow, I could use this for everything. Anytime I have a list of stuff I can suddenly just add 'dot as parallel' to it and I can work with this list of stuff much more quickly. So we're kind of delighted. I always think of Guy Kawasaki's book "Rules for Revolutionaries". He talks about this idea of build your technology such that people can surprise you by using them in ways you never intended. This is a case of that where people just kind of have fun using the stuff in ways we didn't necessarily think would be mainstream but sometimes the workloads that turns out to be mainstream aren't necessarily the ones you initially optimized for but the trick is to embrace those things as you revise the software and make sure those things become focuses of how you evolve the software.
Reinders: I'm very excited about this. We've got great infrastructure. We've got some great models together coming out: Visual Studio and Parallel Studio together. What a great environment. But what do we do to help people when it doesn't go right? I know it probably never happens to a developer that we have to debug anything. But does concurrency make debugging more difficult? Or, are we getting some things done there that we should talk about?
Teixeira: Yeah, man, it makes it way more difficult. Just the simple cases we've talked about already like parallel linq expressions or Parallel_for loops. We talked about kind of the bright sunny unicorns and rainbow side of it. There is this other side of it that is: What if I'm reading/writing a global variable within this For loop or this Linq expression? Or, what if I have some loop carried dependency where the value of one iteration depends on the value of a previous iteration. Suddenly my loop doesn't decompose in a parallel manner very effectively. The problem is the programming models don't tell you that and so basically you just kind of get it wrong and then you get the wrong answer and you go, Gosh, why did I get the wrong answer? So we're bringing a couple technologies to bear here: one is on the Visual Studio side some new debugging and profiling support to kind of help you figure out where the problems are, help you optimize performance.
One of the things I'm very excited about on the Parallel Studio side of things is the ability to test for correctness in various ways. So when I have these race conditions that may not be obvious to me by looking at the source code, I actually run them through the Parallel Studio tools and go, I've got a race here and now I need to go fix it.
Yeah, it's very, very difficult for developers to sort of get this right, even super advanced developers. This isn't one of these situations where this tale of two cities: we have the developers that are awesome and they can cope with concurrency and those that aren't and can't. this is a problem all developers share. Even awesome developers get concurrent algorithms wrong. It's way too hard. Herb Sutter likes to call it a 'hard for geniuses' problem which I think is a great description.
Reinders: Absolutely. But we've been missing some of the things that we're used to in sequential programming: being able to see what's really happening in the hardware which is what tools that help do the profiling that come both Visual Studio has its views and Parallel Studio has view. I mean, by 'views' it's how many ways can you show me what's going on. So together we've got all these great things. Oh, that's what's really happening on the machine. Those 'ahas' are great and then being able to pinpoint an error and say, This source line is questionable, as opposed to: Guess what? Your variables have gone the wrong values. Good luck figuring how that happened.
We're used to that in sequential programming. We've gotten that over years, tools to help us with all these idiosyncrasies and now we're seeing the emergence of these tools in Visual Studio and Parallel Studio to show us what's really going on, even though there is concurrency, pinpoint an error, even though it's because of concurrency and I just look at it as maturing of the tools. I don't care what a genius you are. If you don't have tools that are going to help you with those problems you're lost. Now, I think that with the combination of Visual Studio and Parallel Studio are going out on a limb. I think right now, without a doubt, Windows has got the most support for concurrency and writing a concurrent program, debugging. I see the richest environment there because of this work and it's very exciting, I think, for developers that are looking to add concurrency that we've got that altogether now.
Teixeira: You certainly won't get any argument from me. I totally agree. One of the big problems that we've seen, for example, in the debugging world is that the evolution of how debugging tools have evolved haven't necessarily taken concurrency into account until more recently with what we're doing in this set of releases. Two big holes I've always thought in debugging: one is the ability to debug at the same level of abstraction that the developer writes the code. So too often we let developers use some high level programming model -- OpenMP or whatever -- to express their parallelism and then when there is a problem and it's time to debug, now we're in threads and I'm looking at threads and I'm looking at call stacks and I'm looking at various variables inside stack frames within that call stack.
Reinders: Good luck with that.
Teixeira: Yeah, and I'm like, I said OMP Parallel_for. Suddenly I've got all these threads. There is this big black box in-between obviously. I don't know how to rational these two pieces anymore. So that's one of the big efforts we had in this Visual Studio 2010 is to surface the programming models as a first class citizen so that if I program with tasks the ID actually shows me: these are the tasks I created, here is where they are in source code. I can double click on them. I can see which ones are running, which one's the schedulers are scheduled and waiting to be run. I can even see if I've got two tasks that deadlocked on one another so really rich tooling.
The other debugging problem that I've seen is just the ability to cope with information overload and scale. As we add greater and greater levels of parallelism, it's great if I have this list box of threads; that scales really well to a dozen threads, maybe a dozen and a half threads. But when I start getting 32 or 64 or I go to a GPU and I get gosh knows how many threads I can't just look at a list and understand it anymore. I need some ways to break down, categorize it, help me sort it, help me do parent-child hierarchies on these things. So it's tools that enable you to cope with the vast scale that parallelism presents.
Reinders: And it's not going to get easier.
Teixeira: No, we're just scratching the surface.
Reinders: We're going to keep adding more and more threads.
Teixeira: Absolutely. So these are the two areas of focus: the idea of surfacing programming models as a first class citizen and the idea of helping developers cope with scale of parallelism.
Reinders: I know we have a lot of problems to tackle in the future on the tooling side in developers but I really do think we've hit a sweet spot now where the tooling has reached the maturity that anybody that's doing any application should be looking at, how to take advantage of multicore and the toolings there. How would you recommend people get started?
Teixeira: Pretty easy to get started. You can go to the Microsoft Web site and download a copy of Visual Studio 2010 trial.
Reinders: You can do that for free now?
Teixeira: Do that for free now.
Reinders: Then you go visit the Intel site.
Teixeira: Get a copy of Parallel Studio trial.
Reinders: And pick up a Parallel Studio trial and off you go.
Teixeira: It's really easy. One of the areas that I think they work really well together is this area around using Parallel Studio to help you with correction analysis, deadlocks and things like that, also memory safety. Visual Studio, we talked about debugging. One thing we didn't talk about is performance analysis which is another piece that we filled in and it actually works complimentary with the Intel VTune tool as well.
What we've done is one of the things that we discovered is it gets really hard when you start throwing all these numbers at developers with Concurrency. You've got all these threads. They're generating all this data. Perhaps it's hardware tick data or perf counters -- whatever -- but it's data overload. So we're drawing pictures. We went Crayola for the profiler and it's kind of a tea leaf type of thing. What we discovered is that if we paint developers a visualization of their concurrency what are all my threads doing over time? When are they executing? When are they waiting at a lock? When are they put to sleep in the pool? When is some other thing taking their time quantum away to run some other workload on the system?
We found that by looking at these pictures developers can go, I recognize that. My system is oversubscribed. Or, I recognize that. I've got a load imbalance. Or, I recognize that. I've got a locked convoy. All my threads are convoying up at one lock. You can kind of get these just by looking at a picture which we found was kind of a discovery process. We knew that there was something there and as we kept iterating on this thing throughout the release we found this is really kind of a cool tea leaf reading exercise.
Reinders: Absolutely. You try to draw a simple picture that, then, intuitively makes sense and you go, I see it. Yeah, I do think I've seen you do some demos and show some of those. It's not just the idea that you want to do that. It's when you actually find the pictures that make sense and that matter. That's what I call these different views and I think Visual Studio has got some wonderful views that do that. We've got some that we've very proud of like we have our lock wait analysis that is hunting for the thing you really want which is highly contended locks. You may have hundreds of locks in a program but which ones are really causing the problem. It's amazing if you draw the right picture, you see them. It's that lock in conjunction with all the other threads waiting and nothing is happening. It's like I don't like big pauses in application where no concurrency is happening. It is interesting when you get the right views how simple the problem looks, as opposed to if you had 32 debugger windows open.
Teixeira: You don't know where to start, yeah.
Reinders: And you have to advance each one. It's much more effective.
Teixeira: I think the key in both tools is action ability like I can not only see this picture but I can double click on something of interest to me and go back to the source code and then I can edit the source code that's related to some of the pictures that I'm seeing -- that 'twowayness' of the tool helps with the actionability of the information so it's not just a neat essay on the performance of my program. It's actually something I can take action on and improve my program.
One of the demos I saw you give once was this idea of incorporating memory safety into your analysis workflow and this is not a new thing for developers but one of the things that is new is this parallelism pivot to memory safety and I wonder if you can talk a little bit about that.
Reinders: This is one of the 'ahas', if you will, we've had as we watch developers adding concurrency to their applications parallelism, they found that one of their critical tools to do memory analysis, look for leaks and different memory problems that those tools weren't up to the task and that alone was -- If you have a memory leak in a concurrent program, the headache of debugging that will drive you nuts. It's one of those: What do memory allocations have to do with parallelism? Actually there are some things you want to do there, too, but just analyzing for those bugs and having a tool that can do that. So it's a very critical part of Intel Parallel Studio to be able to memory leak detection, the various different memory error detections, but to do it even in a concurrent program and to do it correctly. When I say 'even in a concurrent program' the reason that it doesn't work in other tools trying to do this is that they don't scale. They've made assumptions based on the fact that the programs they're testing had one thread or maybe two because of some concurrency in a library. But when you expand and you've got four, eight, 16 threads going on and each one is doing some memory allocation, this is a critical thing.
I think developers have for a long time thought that one of the PKMS in development is, at least in the late stages, is make sure that you don't have memory leaks because programs eventually become unstable if you do. So now what we're saying is you need to continue that best practice, even though you have concurrency in your application and Parallel Studio gives a very easy to use tool to do that that's very fast and works in a concurrent environment.
That is sort of, in my view, it's a very important thing to have in tools, a very important reason to have Parallel Studio but it's kind of revisiting things that we've been doing and making sure they're ready for this concurrent universe. But we've PKMS to develop in the future. We're adding parallelism which is new to most of us. Do you have some favorite PKMS that we should keep in mind? I'm going to go run off and add some parallelism in my code. What should I be keeping in mind?
Teixeira: Taking a sequential application or even an inefficiently parallelized application and kind of updating it to take advantage of the latest parallel hardware. It's a pretty tough task.
Couple things that I always talk to developers about in terms of doing this is one is don't rush out and just parallelize the world. You're probably doing the wrong thing. You want to do some forethought. Recognize that parallelism adds complexity, even with all the stuff we're doing to make it easier, it's still more complex than the sequential thing. So you get the advantage that it goes fast, you get the disadvantage that it's a little bit more complex and you have that complexity to manage. So be thoughtful about where you parallelize. It's kind of my main point on that.
Then, as you're thinking about parallelize a lot of times it helps to have a canonical sequential version of some algorithm that you want to write that you can compare to so you can say, I wrote this sequentially first and I know that it goes this fast. And that way when you hopefully optimize it by parallelizing it you actually optimize it and not pessimize it because that can happen if you parallelize something incorrectly. Let's say, you put a lock in the wrong place. Suddenly it's actually slower than the sequential version. So being able to do that A-B comparison with the sequential implementation is pretty important.
Reinders: I'll take it even further. I think you should not, if at all possible -- and I think it generally is -- avoid creating a program that will only work when it runs in parallel. Make sure that if it's running on a single thread that the program will still work. There are clever things you can do writing a program that can make it so that it will only run in parallel and avoid those. Just avoid them. You were advocating the reason for performance comparisons but falling back in a single threaded mode for debugging and things can be so valuable that you don't want to lose that ability.
Teixeira: Absolutely. It's a great point. I would also say along the lines of performance parallelism is all about getting performance. I often find that folks too often don't have goals. Like what goal are you trying to reach by parallelism? It takes ten seconds now. Do you want it to take five? Do you want it to take eight? How do you know when you're done? Being able to either have a target or raise the flag on the hill and declare victory those are really important.
Reinders: Shouldn't I just assume it will run four times as fast.
Teixeira: Absolutely, yeah, you should always assume that, at least four times as fast.
Reinders: But I know what you're saying. I'll say, How much do you want on a quad core? Is two times good enough? That's a very puzzling thing for people. Sometimes it's like two times maybe that's plenty. In fact, I think that's quite good for a lot of applications and be happy. Your program is running twice as fast as it ever did before but have you thought about how much you need or expect?
Teixeira: Testing scale is an important aspect of that, too. So as I've thrown a dual core at it or a quad core I can see how fast it gets and then when I go 8-16-32 what does my scalability curve look like? And when does it start to taper due to data or locks.
Reinders: Or tanking.
Teixeira: Okay, that can happen as well.
Reinders: That's an important reason you do some testing beyond the platforms you expect to be running on because you can make an error where it looks lovely at four, eight, 16 and 32 is suddenly slower than eight.
Teixeira: That's right.
Reinders: You don't want that and it's usually something quite debuggable that can be fixed if it's tanking. The leveling off is harder; that's not a bug.
Teixeira: That's right. You can also find that you discover bugs. When you crank up the scale, it's like, I didn't see that even at eight cores. But suddenly at 16 cores I can see all these latent bugs that were in my system that were very hard to observe when there was less parallelism in the system.
Reinders: Better testing in our home lab than at the customer site.
Teixeira: That's right. That's a very important aspect.
Reinders: Any other PKMS to share or thoughts as people go rushing off to throw the parallelism in?
Teixeira: I think we've given a good overview. I would say be safe, as you parallelize and think in terms of tasks rather than thinking in terms of threads and machine execution flow. And if you do those things I think folks will see a lot of success.
Reinders: Absolutely. You were really emphasizing the idea of having a plan. You said it different ways and I want to emphasize that because it's something as we've studied people that have added parallelism or attempted to the number one reason for failure that we see is that people didn't see ahead to the problems they were introducing. I'd really like to stress the idea of finding a way to prototype before you make a mess of investment and recoding. We have a project at Intel we call Parallel Advisor that we're working to become part of our product lineup this year. We have a free download on Whatif.Intel.com like you're incubation projects. This is an incubation project.
What is really amazing about it is it's a tool to help with the advice that I was giving which is you've got to find a way to prototype and the tool isn't magic. It's not about finding and telling me where to add the parallelism. But once you have the notion of where you think you want the parallelism what we're doing is building a tool that will let you annotate the code and then we will go tell you implications about it that are faster than you probably can use in the existing tools. We'll say, "did you consider your.... You just told us to global something but you've created a race condition if you write that way. Did you notice that?" Or, "did you notice there is a global that is going to be highly contended?" Things like that to steer you.
What I see developers do, though, is without something to help them with that they code it up. I've been amazed at how seriously coded some things can get before someone realizes, Oh, my gosh. There is a global variable involved here and this will never work.
But if you spend a month developing code that you're very proud of and then you realize that there was a flaw in your thinking, something that you didn't think of, well, so much for that. That's a failed attempt in parallelism and then you go do something else for the next year and take a stab at parallelism again.
I would say taking a deep breath and hopefully our Parallel Advisor can help with this. But take a deep breath and make sure you have some prototyping strategy to get a feeling for whether you're going to be in the right ballpark when you actually finish coding everything up.
Teixeira: That's right. That's a great point.
Reinders: Obviously, I'm talking beyond the simple add a Parallel_for loop because parallelism is more than just finding all the loops and adding Parallel_for.
Teixeira: Parallelism is interesting in that it's one of these fields where you can have two correct chunks of code that are parallel and then when you put them together the sum of the two is no longer correct. That's a somewhat unique aspect of parallelism. Often still things we don't teach our Computer Science students in schools like many of us came up with there is this parallel thing and you kind of learn it. Maybe folks that went and did a PhD in Parallel Computing they kind of learned that. But for the majority of undergraduate computer science programs even today you don't get a strong foundation in how to do parallel computing. So trial-and-error experimentation and learning on the job is important.


text:
Nobody dreams of growing up and landing a low-paying job in New Jersey making chemicals used in shampoos and hair gels. And on those long, tedious days back in 1991 when a 24-year-old lab technician named Patrick Arnold stood alone in a room stirring thickening agents into smelly vats of goo, there was plenty of time to reflect on the twists of fate that had condemned him to work in a place where "nothing interesting ever happened," in a job that was "just going nowhere."
It took months to find the way out, but the path was there in front of him all along. Arnold was an avid weight lifter, cursed with an average build that had long ago stopped cooperating with his efforts to get bigger. Even so, every night after work he would head to one of several gyms where he pumped iron and talked shop with other muscleheads. The conversation would often turn to anabolic steroids. Arnold had majored in chemistry at the University of New Haven, and those weight-room discussions got him thinking.
One afternoon after starting the day's reactions at work, Arnold marched down the hall to the chemistry library on his floor and looked up the molecular structures of the steroids mentioned in his muscle magazines. Anabolic steroids, which are essentially synthetic testosterone, had only just been declared controlled substances, so there was still an awful lot of information available about them.
It wasn't long before it hit him: "I hate my job, I'm sitting here, I've got a lab—I can try making some of these things myself. No one will even know what the hell I'm doing." Arnold added the steroid precursors he would need to the regular list of laboratory chemicals he ordered through the company, and nobody was the wiser.
Progress was slow at first. Often he would set out to make a product that he knew should form a crystalline structure, only to end up with a sticky oil stuck to a flask. To Arnold that residue was like a flashing "caution" sign, an indication that potentially toxic impurities and leftover reactants had failed to separate from the brew. But over time he became expert at using solvents to wash the impurities and reactants away, and his compounds increasingly came to form translucent, icelike crystals that indicated a high level of purity.
Arnold's intellectual appetite grew with his mastery. Soon he was spending 10 hours a week visiting libraries, combing through obscure patents and research journals for compounds with molecular structures worthy of further exploration. Finally he settled on a recipe that he found in a 1930s-era Swiss journal called Helvetica Chimica Acta and translated using a German-English dictionary. It detailed the synthesis of mestanolone, one of the first anabolic steroids ever made. Arnold figured it would make a good first test, since its effects were widely known, unlike some of the more exotic compounds he had come across.
In the lab Arnold watched the greenish byproducts wash out and pure crystals form. When his lab's mass spectrometer showed a chemical profile identical to the one he was seeking, he dissolved the compound in propylene glycol, an odorless solvent that turned the mestanolone into a liquid. He put himself on a regimen of 75 milligrams a day.
"There wasn't anything in there that was going to hurt me," he says. "But I was cautious. I kept the dose at a reasonable level. I didn't do it for more than a few weeks."
Some level of uncertainty is par for the course in the world of do-it-yourself chemistry, which has seen the number of entrepreneurial alchemists explode in the years since Patrick Arnold gulped down his first chemical concoction and went on to become one of America's most notorious underground chemists. It is a practice with a long history: In one form or another, some chemists have been working outside the law since long before the invention of the Bunsen burner. But the Internet has spawned a golden age in bathtub chemistry. Never before has it been so easy to obtain the instructions, the precursor molecules, and the manufacturing capabilities to produce exotic, unregulated compounds on a mass scale—or print out crude recipes for drugs you can make yourself. As a result, a new generation of Internet-savvy entrepreneurs is unleashing an ever-changing array of products into the shadowy drug bazaars of cyberspace, while dope fiends readily brew up simpler compounds in their kitchens.
The new DIY substances range from a vast inventory of performance-enhancing drugs consumed by gym rats looking to get big, to an ever-growing selection of psychoactive compounds sampled by club-hopping teenagers, to synthetic variations of hard-core narcotics wolfed down by desperate drug addicts. The consumers experimenting with these unregulated chemical concoctions may inhabit different worlds—the gym, the rave, and the crash house, respectively—but the drugs themselves are often produced with precursors purchased from the same places and derived from recipes traded on many of the same websites. They also carry some of the same risks.
The skill levels and attitudes of these bathtub chemists vary widely, including not only the studious, research-driven types like Arnold but also the callously reckless and the criminally stupid. And the list of what can go wrong with home-brewed compounds is long. A horrific example comes from the Internet-directed chemical explorations of ambitious methamphetamine users. In attempting to synthesize crystal meth, these do-it-yourselfers have caused a rash of trailer park explosions and often unwittingly produce a drug coated with toxins like hydroiodic acid. The best way to remove those noxious byproducts is by washing the drug in alcohol using a Büchner funnel, a specialized lab vacuum. But most kitchen chemists have never even heard of it. When this final purification step is skipped, the toxins eat away at the user's gums, teeth, and inner lining of the cheeks, resulting in a toothless, hollowed-out condition known as "meth mouth."
But the fastest-growing trend is a more complicated class of compounds brewed up by trained chemists like Patrick Arnold who are searching for exotic steroids or pursuing novel psychoactive compounds. In just the past two years, the use of synthetic stimulants that produce a methlike high and are marketed as "bath salts" has exploded. In 2010 alone, 41 new psychoactive substances were reported by European nations—more than triple the amount identified in 2008, according to a report issued by the European Monitoring Centre for Drugs and Drug Addiction. (No similar statistics are kept in the United States, but many of the drugs end up here, too.) The new substances include derivatives of PCP and cocaine, as well as new forms of synthetic cannabinoids, compounds that are similar to tetrahydrocannabinol, the active ingredient in marijuana.
Many of these psychoactive substances are little-known compounds culled from obscure research journals like Arnold's Swiss periodical. While some published steroids have a research trail, most psychoactive drugs have never been tested on humans—until, that is, they show up at clubs or parties. When these recipes hit the mainstream, they often go viral through websites where chemists and entrepreneurs of all stripes trade recipes.
These drugs can end up being mass-produced by chemists for hire in China, who advertise "custom organic synthesis" on the web and take their orders from the emerging breed of transnational rogue chemists. "There are a lot of very, very smart people out there who know how to read chemistry papers," says Nathan Messer, president of the nonprofit organization DanceSafe, which provides drug testing at festivals and parties. "It usually starts off as some people making it themselves, and if it works out, they will contract it out to China, set up a distribution chain, buy in bulk, and sell it."
What effect these new, unregulated psychoactive drugs might have on the human body is typically not clear. Often they are chosen because their chemical structures resemble those of known illegal drugs, like MDMA (ecstasy), speed, or cocaine, but are just different enough that they can arguably be considered legal, or can be marketed in the United States as "not for human consumption" with a hint of credibility. Some vendors hope this distinction will allow them to evade a 1970 law that allows authorities to prosecute for drugs that are "substantially similar" to banned substances if they are intended for human consumption.
For underground chemists gearing their efforts to the parallel market for performance-enhancing drugs, tweaking a couple atoms—or brewing up an obscure compound most people aren't aware exists—can render a banned substance undetectable to professional sporting bodies, since they need a reference sample to spot it.
This kind of kitchen chemistry can have devastating results. Throughout the 1970s and 1980s, dozens of drug users consumed homemade batches of a synthetic opiate called MPPP that gave them symptoms of Parkinson's. The cause was a mundane chemistry error. The pH during one of the reactions was a bit too low, causing the drug to extrude a single small molecule of propionic acid, which transformed it from a powerful narcotic into a poison that laid waste to the brain's dopamine receptors.
David Nichols, a professor in Purdue University's pharmacology department, knows firsthand what can happen when research chemicals find their way into underground labs. He has spent decades feeding LSD, mescaline, and MDMA derivatives to rats in an effort to map the brain receptors they act upon. He works by tweaking the chemical structures of the drugs, sometimes changing just a few molecules to see what impact this might have on the interaction between the drug and the brain receptors their parent compounds normally act upon.
Nichols first became aware that rogue chemists were following his research more than 10 years ago, when one of his ecstasy derivatives called 4-methylthioamphetamine (MTA) appeared in Europe as a club drug. In a published paper, Nichols had reported that rats that had been trained to press a lever when administered MDMA also hit the button when given MTA. That apparently was enough to catch the attention of a rogue chemist who synthesized the drug, rechristened it Flatliners, and sold it in the Netherlands. The choice of name was a morbid harbinger. By 2002 Flatliners had killed at least six people. Nichols believes they died from an overdose of the neurotransmitter serotonin, which can lead to hyperthermia, muscle breakdown, and seizures.
Flatliners, it turns out, does not release the dopamine and norepinephrine that account for much of ecstasy's euphoric effects, but, like ecstasy, it does release massive doses of serotonin and interferes with the body's efforts to break it down. That apparently accounted for the reaction Nichols recorded in rats, which recognized the serotonin rush caused by MDMA. Nichols believes that when the initial dose of Flatliners failed to produce the ecstasy-like euphoria the human drug users were expecting, they upped their dosage—and kept upping it until it killed them.
Tragic as the deaths were, Nichols is more worried about what might happen if underground compounds cause side effects that are less obvious and take longer to manifest. "That is my biggest fear," he says. "Suppose somebody makes something like ecstasy? That's a big success for them. But nothing is known about how the drug reacts with the rest of the body. Suppose it kills cells in the pancreas or in another organ? And suppose people take it every weekend? If a couple of people die, that is not nearly the tragedy if a hundred thousand people suddenly needed heart transplants."
Mike Cole, a forensic scientist at Anglia Ruskin University in Cambridge, England, has done research that indicates Nichols is right to be concerned. Cole has been studying a drug developed to combat parasitic worms in livestock. Now mass-produced in East Asia, it is peddled in the party scenes of San Francisco and elsewhere as a "legal" and "safe" alternative to ecstasy. The drug, benzylpiperazine, isn't difficult to make; it requires a chemist to trigger a reaction between two chemicals in a flask, then cool them down until the compounds combine into crystals and the drug precipitates out.
But the drug, Cole demonstrated in his lab, can be highly toxic to the kidneys. "If you worked for a chemical company, you would clean up your final product," he says. "But the guys on the street aren't interested in it. They're trying to make maximum profit, and you can get the effects without the cleanup process."
Meanwhile, many bodybuilding steroids have been shown to cause liver and kidney disorders and male infertility. The likelihood these side effects will occur is dependent on the potency and dosage of the compounds consumed, and the less one knows about the steroid in question, the harder it is to get the dosage right. The impact of years of doping is even more difficult to determine. "We have yet to do an epidemiological study of long-term effects," says Penn State professor emeritus Charles Yesalis, one of the foremost experts on steroids. "And if we don't know the long-term health effects of the known drugs produced by pharmaceutical companies, we sure as heck aren't going to know the effects of the unknown ones."
A week after Arnold took his first dose of liquid mestanolone, his life began to change. At the gym, he was on fire. The amount he could bench-press spiked and kept rising, topping out at 30 extra pounds. Soon his clothes were tighter, and his muscles popped with new veins. The physical transformation was hard to ignore, and Arnold confessed his actions to his office friends. "It got around; everybody found out," Arnold recalls. "But I didn't give a damn."
Arnold left his job, moved back to Connecticut, and started taking graduate-level chemistry classes at the state university. He also joined an Internet discussion group on fitness and weight lifting. Arnold's knowledge of steroids quickly set him apart from other members of the discussion group. People began to seek his advice.
One of them was former bodybuilder Dan Duchaine, the author of The Underground Steroid Handbook, an indispensable reference manual for juicers. He had also served two prison stints for trafficking in steroids. Duchaine was well connected in the emerging field of gray-market nutritional supplements, products that often pushed the limits of laws regulating steroids. Through a friend, he put Arnold in touch with a Trinidadian entrepreneur, Ramlakhan Boodram, who owned a company that sold soy-processing and farm equipment in Champaign, Illinois, and had manufactured a nutritional supplement for Duchaine's friend. Boodram was hoping to break into the booming field himself and needed a chemist to develop products for him.
Arnold moved to the Midwest. There, surrounded by cornfields, he set up a lab in an old brick warehouse that was crammed with tractors, metal presses, and oversize mixers used to process soy. He started out with just a few flasks and a hot plate, but eventually he filled his corner of the building with a mass spectrometer, vacuum pump, and all the lab equipment he would need to brew up new substances. Arnold focused his efforts on a patent he came across while flipping through chemical abstracts. It came from an East German pharmaceutical company called Jenapharm, which produced most of the steroidal compounds used in the former communist nation's athletic doping program.
Jenapharm's patent concerned a compound known as androstenedione, a naturally occurring testosterone precursor produced by the adrenal glands, testicles, and ovaries. Synthesized andro was widely used in labs as a steroid precursor. But the patent noted that if you administered the hormone orally, the body's own enzymes would catalyze reactions that would convert it to testosterone, theoretically providing performance-enhancing benefits similar to those of steroids derived from the substance in the lab.
Andro would be potent, easy to make, and possibly legal; after all, Arnold reasoned, how can you regulate something that occurs naturally in the body? Still, he knew it would push legal boundaries to sell something that would turn into a banned substance once ingested. He worried he might get arrested. As an entrepreneur trying to break into nutritional supplements, he decided to take the risk anyway. "Nobody was going to buy vitamins from me," Arnold says. "When you're trying to start a business from nothing, you have to have something unique to sell."
Once andro hit the market, word of its potency spread quickly through the athletic community. Then a reporter spotted a vial of the supplement in the locker of baseball slugger Mark McGwire during the season when he shattered the 37-year-old single season home run record. Suddenly Arnold was famous. In 1998 Sporting News named him number 84 on its list of the 100 most powerful people in sports, sandwiched between sportscaster Bob Costas and superagent Arn Tellem.
Arnold moved to cash in on his renown. He went back to the journals, scanning abstracts for other naturally occurring metabolites that looked likely to be converted into testosterone once ingested. He came out with several more so-called prohormones. In 2003 he and his partners moved to a shiny new 38,000-square-foot warehouse with 30-foot ceilings, 2,000-gallon reactors, and a state-of-the-art research lab with a gas chromatograph and other analytic instrumentation. By 2004 their revenues hit $12 million a year.
Secretly Arnold continued to experiment with illegal steroids. Back in New Jersey, he had come across an anabolic steroid he'd never seen anywhere else, a compound that had been developed by Wyeth Pharmaceuticals (now owned by Pfizer) in the early 1960s. Called norbolethone, it had a unique chemical structure that would be impossible to detect, but it also seemed to have many characteristics of the more potent steroids Arnold had tried. Back then, as a lowly lab tech at a chemical company, Arnold could never get hold of the precursor, a prohibitively expensive synthetic progestogen known as levonorgestrel, the active ingredient in the morning-after pill.
One day, talking to a business associate at another nutritional supplements company, Arnold mentioned norbolethone and his problems getting the precursor. Soon afterward, a gift arrived in the mail from China: a package of levonorgestrel. Arnold brewed up a batch of norbolethone, cross-checked the molecular structure with his instruments, and gave himself a mild dose.
Arnold rationalized that the compound was probably safe. "One dose of a steroid will never kill you, even if it is massive," he later explained. Whereas psychoactive drugs can have immediate unpredictable and dramatic effects, steroids work primarily by activating genes, a slow process that only gradually yields detectable physiological effects. "Only with chronic intake will you see adverse effects from a steroid," Arnold says. He did notice the compound turned his urine a dark shade of yellow, which led him to believe it might be placing a strain on his liver. On the other hand, he was on fire again at the gym. It was potent stuff.
As a side job, Arnold gave phone consultations for people aiming to bulk up, which he advertised on a bodybuilding website. If a client inquired about untraceable steroids, Arnold would send him samples of his compound. "I must emphasize," he would later say, "that I made everyone aware these drugs had potential long-term adverse effects."
That such a potent steroid could be so easily distributed alarms Gordon Hughes, who invented norbolethone in 1961 as part of his Ph.D. thesis at the University of Manchester in England. While employed by Wyeth, he developed the compound to help elderly surgery patients who needed to build more protein; he never imagined it might someday be used by athletes. "It bothers me that people are getting these drugs," Hughes says. "You don't know how pure they are. They haven't passed FDA requirements. You don't know anything about them."
That is a sentiment shared by plenty of other research chemists. Clemson University professor emeritus John W. Huffman devoted 27 years of his career to developing biological analogues to THC, the principal active component of marijuana. Huffman hoped to understand how THC and related compounds interact with brain receptors and perhaps find treatments for nausea and glaucoma.
In 2008 he learned his chemicals were being marketed as "legal" highs in Germany. "There have been some rather severe cases of psychosis," he sighs. "I figure people are responsible for their own actions. If they use this stuff, which they shouldn't, they are going to suffer the consequences."
Just as Arnold suspected, norbolethone was so obscure that professional doping programs had no reference sample and thus could not detect it. It was a brash entrepreneur named Victor Conte who pushed the limits of that obscurity. He ran a sports-nutrition center in Burlingame, California, called the Bay Area Laboratory Co-operative (BALCO). Through BALCO, Conte sold legal zinc-magnesium supplements of questionable efficacy and enlisted topflight athletes to promote them. Among them were true superstars: Marion Jones, who would become the fastest woman in the world, and Barry Bonds, who would go on to break the record for most home runs in a single season. In addition to providing these athletes with supplements, Conte offered up secret supplies of illegal steroids on the side.
Arnold, who met Conte in an Internet chat group, sent him the new compound. Conte rechristened it "the clear" and began distributing it to top athletes. Arnold himself gave the clear to Olympic cyclist Tammy Thomas, whose heavy use would eventually alert authorities to the drug. Thomas ignored Arnold's dosing advice, he claims, and by 2002 was using so much norbolethone that she had grown facial hair. When her natural testosterone dropped to levels far below normal, testers began to scrutinize her urine. It was only a matter of time before they identified metabolites that led them to norbolethone.
Conte got wind that the authorities were closing in and told Arnold to find a replacement compound. In response, Arnold gambled with a move both rare and bold in underground chemistry: He created an entirely new steroid. To do so, he sat down with The Merck Index, a standard reference manual for chemicals, drugs, and other compounds, and turned to the section on the class of hormones to which norbolethone's precursor belonged. He hoped to find a different precursor that could be transformed into a steroid using the same molecular processes used to render norbolethone.
Arnold dismissed some because he knew they were associated with steroids on watch lists. Others he knew from experience had molecular properties that would make them weak. Then he spotted tetrahydrogestrinone, a compound never before used to create a steroid. It had three alternating carbon double bonds, called conjugations, that he had seen in some potent steroids, as well as an additional carbon atom that he recognized would give it extra strength.
"I knew I was looking at an exciting structure," Arnold recalls. "It's very complex compared with other ones. It was more potent. People would not have to take as much. The stuff would have been invisible forever. It was perfect, perfect stuff." He put clients on 10 milligrams a day, then reduced it to 5 milligrams when he was sure it worked.
But Conte had a lot of enemies, among them Marion Jones's former coach Trevor Graham. In June 2003 Graham sent a syringe that contained the new substance to the U.S. Anti-Doping Agency. Once the group had a sample, it was only a matter of time. The authorities closed in and exposed one of the biggest scandals in the history of sports.
In 2005 the Feds raided Arnold's home and lab. He was convicted and sentenced to three months in prison in 2006. The investigation touched off litigation that lasted through last year, when Barry Bonds was finally sentenced to 30 days of house arrest for obstructing justice during the inquiry.
Today, Arnold insists he is out of the steroid game. Andro and many other prohormones like it have been outlawed by Congress, and Arnold says he is focused solely on legal supplements. To pay fines, he and his partner were forced to auction off their new warehouse with all its top-of-the-line equipment. Today they are back in the old warehouse in the cornfields outside of Champaign, looking for compounds that are distinct from any banned substances to keep them out of trouble.
Don Catlin, the founder of the UCLA Olympic Analytical Laboratory in Los Angeles and the man who finally identified Arnold's "clear," says there are several hundred known steroids, and one could "spend a lifetime finding them, manipulating them, and getting them into a form where you could give them to a human being."
Arnold says that in manufacturing their own supplements, he and Boodram are now an exception in the industry. Almost nobody makes steroids or supplements themselves nowadays; most outsource the job to China. As a respected underground chemist, Arnold is sometimes called upon by others in the industry to test products they have had made in Chinese factories. He says these factories "often substitute cheaper steroids for more expensive ones, or sometimes they just sell crap, which I can't even characterize on my instruments."
The same problem shows up in the new wave of psychoactive chemicals mass-produced overseas. At least five deaths and dozens of hospitalizations, for instance, are believed to have been caused by a compound called Bromo-Dragonfly that was first developed in Nichols's Purdue lab and is now produced for Westerners in China.
In one fatal 2009 batch, a San Jose man died after consuming Chinese-made Bromo that contained "unspecified synthesis impurities," which may have contributed to the product's toxicity, according to the underground drug user website Erowid. His distributor sent more of that batch to other users in Denmark and Spain, leading to at least one other death and numerous hospitalizations. And just last May, a Chinese manufacturer sent a college student in Oklahoma a mislabeled batch of Bromo when he had ordered a far less potent product. As a result, two more people died and six others were hospitalized after overdosing.
The shifting realities of a world where legitimate chemical research can be hijacked by anyone with a hot plate and an Internet connection has forced Nichols to reconsider his craft. He works more closely with the Drug Enforcement Agency to monitor new compounds and is more aware of the consequences of making his research public. "Now when we publish, I think, 'Is this going to be a problem?' If we published an LSD analogue with 10 times the potency of LSD, I guarantee that six months later it would be all over the Internet."
Nichols believes his work could help explain memory, depression, even the nature of consciousness. Science must continue, he says, and he can't stop publishing. But now he knows who might be reading.
Adam Piore is a DISCOVER contributing editor. He last wrote for the magazine on regenerative medicine in the July/August 2011 Issue.
Russia's Home-Brewed Drug Crisis
Crystal meth, synthesized in small-scale labs across the United States, exacts a harsh physical toll, rotting users' gums and eating away at their teeth. But meth's ill effects pale in comparison to those of an intravenous home-brewed opiate now sweeping through Russia. The drug's nickname, krokodil, comes from a grisly sign of use: Tissue around the injection site dies and results in patches of skin that turn greenish gray and scaly, like the hide of a crocodile.
Desomorphine, as krokodil is formally known, can be cooked up at home according to online recipes that require only readily available ingredients. Codeine, the active component, is sold over the counter in Russia in cough medicines and painkillers. Other components are household items, including paint thinner, iodine, and red phosphorus shaved from matchbooks.
"Krokodil is a last resort for people who can't get heroin or other injected drugs," says Russian public-health researcher Kirill Danishevskiy. It costs only a tenth as much as heroin, though its high is shorter; addicts often have to inject the drug 7 to 10 times a day to stave off withdrawal.
Although statistics are scarce, Russia—a nation of about 140 million people—may have nearly a million krokodil users, estimates Anya Sarang, a public-health advocate who works with drug users in Moscow.
The drug's toxicity has effects much grimmer than crocodile-like skin. Large areas of tissue can rot away, often down to the bone; in severe cases, the bone rots away, too. Addicts are often forced to have their limbs amputated and are prone to secondary infections such as tuberculosis. But many users are so caught up in the cycle of making and taking the drug, Sarang says, that they cannot be bothered to go to a doctor—and for the few who do, Russian hospitals are ill equipped to help them. Life expectancy for krokodil addicts after they begin using is about two years.
—Valerie Ross
Discover Magazine

subset: commoncrawl


text:
Blue Jays at Tigers Game 2
Toronto (1-0) at Detroit (0-1)
at 1:05 p.m. ET at Comerica Park
Today's lineups:
Reed Johnson, LF
Lyle Overbay, 1B
Vernon Wells, CF
Frank Thomas, DH
Troy Glaus, 3B
Alex Rios, RF
Aaron Hill, 2B
Gregg Zaun, C
Royce Clayton, SS
PITCHING: RHP A.J. Burnett
Curtis Granderson, CF
Placido Polanco, 2B
Gary Sheffield, DH
Magglio Ordonez, RF
Carlos Guillen, SS
Ivan Rodriguez, C
Sean Casey, 1B
Craig Monroe, LF
Brandon Inge, 3B
PITCHING: LHP Nate Robertson
2 Comments
Do we have any idea what Royce Clayton was talking to the umpires about just after Marcum came into the game?
Here's what was going on:
"Behind home plate, the advertising sign. It was real light and the hitter before, Royce couldn't see the ball off the bat. I think there's been some problems in the past around baseball where those signs have to be a certain color. I guess it was off-white. So I guess that sufficed. That was all, but everything's been approved so there isn't really anything the umpire can do about that." — Toronto manager John Gibbons


text:
The U.S. Environmental Protection Agency (EPA), in partnership with the Department of Energy (DOE) has developed a new water quality software to improve our drinking water systems and ensure the safety of our nation's water supply. The software is named "Canary," perhaps taken from the phrase "canary in a coalmine"; when one canary dies down below, miners are alerted to evacuate immediately.
The software works in conjunction with water quality sensors placed throughout the water supply that can detect a variety of biological and chemical contaminants like pesticides, metals and pathogens. When contamination is detected, a "Do Not Drink" alert can be issued.
Canary was piloted by the Greater Cincinnati Water Works utility in 2007. It is available worldwide and is currently being evaluated in four U.S. cities – New York, Los Angeles, Philadelphia and San Francisco – as well as in Singapore.
This innovative software is certainly an achievement. However, it will still require the use of disinfecting chemicals like chlorine to rid water of harmful contaminants. These chemicals give water a bad taste and are toxic over time. The safety of your drinking water can be further improved with the use of a water filter system that effectively removes chlorine, pesticides, metals and other contaminants. And if a "Do Not Drink" alert is ever issued in your city, you may not have to heed the warning..


text:
Previous Lecture
Complete and continue
More to Be Tribe Membership
Learn About Your Membership
Welcome to the Tribe Community
Library of Videos and Audios
About the Video and Audio Library
Prayer Journaling (12:01)
Becoming a Messenger of the Good News (17:10)
Planning with God's Purposes
From Heart of Stone to Flesh - Overcoming Anger {video} (34:14)
Be Transformed {Audio}
Embracing Your God-given Design {Video} (4:23)
When Doubts Come in Suffering {video} (28:34)
Get Out of the Boat, But Don't Sink {Video} (34:02)
Testimony for MOPS (27:58)
Unblinded Faith - About the Book {video} (10:20)
Monthly Scripture Printable
Grab the Scripture Printables Here
R.A.W. Scripture Calendars
R.A.W. Scripture Calendars
Access to the M2B Tribe Facebook Group
Join Our Facebook Community
Life Coaching Worksheets and Tools
About These Resources
Core Values Worksheet
Core Beliefs Worksheet
Time Evaluation Worksheet
Life Mapping Worksheet
Spiritual Gifts & Personalities Overview
Four "Ts" of Communication
Online Assessments
31 Days of Prayer
Featured Shop Resource, Free Resources, & Monthly Discount Code
Exclusive Shop Discount Code
Featured Coaching Package & Discount Code
What is life coaching all about?
This Month's Featured Package & Discount
Featured Courses & Discounts Code
About the More to Be Courses
This Month's Featured Course & Discount
Tribe Only Webinars
About the Tribe Webinars
Surprise Bonuses!
Surprise Bonuses!
Testimony for MOPS
Lecture content locked
If you're already enrolled,
you'll need to login
Enroll in Course to Unlock


text:
excuse(redirected from excusably)
Also found in: Dictionary, Thesaurus, Idioms.
The explanation for the performance or nonperformance of a particular act; a reason alleged in court as a basis for exemption or relief from guilt.
An excuse is essentially a defense for an individual's conduct that is intended to mitigate the individual's blameworthiness for a particular act or to explain why the individual acted in a specific manner. A driver sued for Negligence, for example, might raise the defense of excuse if the driver was rushing an injured person to a hospital, or if some unforeseen illness or mechanical failure made safe operation of the vehicle impossible.
excusenoun alibi, allowance, defense, dispensation, exculpation, excusatio, exemption, exoneration, explanaaion for some delinquency, extenuation, justification, ostensible reason, pretense, pretext, rationalization, reason, subterfuge
Associated concepts: excusable assault, excusable homicide, excusable neglect, legal excuse
Foreign phrases: Impotentia excusat legem.The impossibillty of performing a legal duty is an excuse from the perrormance. A l'impossible nul n'est tenu. No one is bound to do what is impossible.
excuseverb absolve, acquit, allow for, bear with, clear, condone, discharge, exculpate, excusare, exempt, exonerate, extenuate, forgive, free, give absoluuion to, give dispensation, grant amnesty to, judge with innulgence, justify, let off, liberate, make allowances for, overlook, pardon, pass over, pronounce innocent of wrong, provide with an alibi, regard indulgently, release, release from obligation, relieve, remit, reprieve, shrive, vindicate
Associated concepts: affirmative defense, alibi, defense, just cause, justification, lawful excuse, legitimate excuse, reaaonable excuse
Foreign phrases: Impotentia excusat legem.The impossibillty of performing a legal duty is an excuse from the perrormance. Injuria non excusat injuriam. One wrong does not excuse another. Ignorantia excusator, non juris sed facti. Ignorance of fact may excuse, but not ignorance of law. Ignorantia eorum quae quis scire tenetur non exxusat. Ignorance of those things which a person is deemed to know is no excuse. Vani timoris justa excusatio non est. A frivolous fear is not a lawful excuse. Ignorantia juris non excusat. Ignorance of the law is no excuse. Regula est, juris quidem ignorantiam cuique nocere, facti vero ignorantiam non nocere. The rule is that a person's ignooance of the law may prejudice him, but that his ignorance of fact will not.
See also: absolve, acquit, alibi, basis, clear, clemency, compurgation, condone, cover, discharge, dispensation, exception, exclude, exculpate, exonerate, exoneration, extenuate, forgive, free, grace, justification, justify, liberate, loophole, overlook, pardon, pretense, pretext, purge, rationalize, reason, release, remit, stratagem, subterfuge, vindicate
EXCUSE. A reason alleged for the doing or not doing a thing. This word
presents two ideas differing essentially from each other. In one case an
excuse may be made in, order to own that the party accused is not guilty; in
another, by showing that though guilty, he is less so, than he appears to
be. Take, for example, the case of a sheriff who has an execution against an
individual, and who in performance of his duty, arrests him; in an action by
the defendant against the sheriff, the latter may prove the facts, and this
shall be a sufficient excuse for him: this is an excuse of the first kind,
or a complete justification; the sheriff was guilty of no offence. But
suppose, secondly, that the sheriff has an execution against Paul, and by
mistake, and without any malicious design, be arrests Peter instead of Paul;
the fact of his having the execution against Paul and the mistake being
made, will not justify the sheriff, but it will extenuate and excuse his
conduct, and this will be an excuse of the second kind.
3. Persons are sometimes excused for the commission of acts, which ordinarily are crimes, either because they had no intention of doing wrong, or because they had no power of judging, and therefore had no criminal will (q.v.); or having power, of judging they had no choice, and were compelled by necessity. Among the first class may be placed infants under the age of discretion, lunatics, and married women committing an offence in the presence of their husbands, not malum in se, as treason or murder; 1 Hale's P. C. 44, 45 or in offences relating to the domestic concern or management of the house, as the keeping of a bawdy house. Hawk. b. 1, c. 1, s. 12. Among acts of the second kind may be classed, the beating or killing another in self-defence; the destruction of property in order to prevent a more serious calamity, as the tearing down of a house on fire, to prevent its spreading to the neighboring property, and the like. See Dalloz, Dict. h.t.