text:
|April, 2009
|Doogue et al.
|May, 2008
|Taylor et al.
|Test pad structure, a pad structure for inspecting a semiconductor chip and a wiring subtrate for a tape package having the same
|December, 2009
|Lim et al.
|March, 2009
|Kushiya et al.
|Magnetic resonance reception coil composite structure
|September, 2006
|Xue et al.
|Antistatic Demonstrator
|November, 2008
|Mouhot et al.
|Electricians phase tester
|May, 2007
|Measurement of pipe wall thickness using magnetic flux leakage signals
|October, 2009
|Dutta et al.
|Linear inductive fluid level sensor
|May, 2002
|Strayer et al.
|May, 2009
|Garland et al.
|Surface to underwater viewing device attachment for metal detector
|August, 2004
Not applicable.
The present invention relates to probe assemblies of the type commonly used for testing integrated circuits (ICs) that are fabricated on a wafer or substrate.
The trend in electronic production, particularly in integrated circuit technology, has been toward fabricating larger numbers of discrete circuit elements with higher operating frequencies and smaller geometries on a single substrate or "wafer." After fabrication, the wafer is divided into a number of rectangular-shaped chips or "dies" where each die presents a rectangular or other regular arrangement of metallic bond or contact pads through which connections are made for the inputs and outputs of the electrical circuit on the die. Although each die is eventually packaged separately, for efficiency sake, testing of the circuits formed on the wafer is preferably performed while the dies are still joined together on the wafer. One typical procedure is to support the wafer on a flat stage or "chuck" and to move the wafer in X, Y and Z directions relative to the head of a probing assembly so that contacts on the probing assembly move relative to the surface of the wafer for consecutive engagement with the contact pads of one or more of a plurality of dies or test structures on the wafer. Respective signal, power and ground conductors that interconnect the test instrumentation with the contacts on the probing assembly enable each circuit on the wafer to be sequentially connected to the instrumentation and tested.
Gleason et al., U.S. Pat. No. 5,914,613, discloses a membrane probing system for use in a probe station. The membrane probing system comprises a probe head and a membrane probing assembly. The probe head includes an interface board, a multi-layer printed circuit board that facilitates interconnection of the membrane probing assembly and the test instrumentation supplying power and signals to and receiving signals from the electrical circuit being tested, the device-under test (DUT). The power and signals are transmitted over one or more conductors that are conductively interconnected with respective data/signal traces on the interface board. The data/signal traces on the interface board are conductively connected to respective conductive traces on the surface of the membrane assembly. A metallic layer below the surface of the interface board provides a ground plane for the interface board and a ground reference for the power and lower frequency signals.
Typically, higher frequency signals; commonly in the radio or microwave frequency ranges, collectively referred herein to as RF signals; are communicated between the test instrumentation and the membrane probing system with coaxial cable. The coaxial cable is connected to an adapter that is secured to the interface board. A second portion of coaxial cable, conductively interconnected with the first portion in the adapter, is connected to one or more conductive traces on the surface of the interface board. Typically, the end of the second portion of coaxial cable is cut at an angle and the conductors of the cable are connected to respective traces on the interface board to transition the signal path from the coaxial cable to a co-planar waveguide. For example, the center connector of the coaxial cable may soldered to a trace on the interface board while the outer conductor of the cable, connected to a ground potential, is soldered to a pair of traces that are respectively spaced apart to either side of the trace to which the center conductor is connected transitioning the signal path from coaxial cable to a ground-signal-ground (GSG) co-planar waveguide on the interface board. The traces on the interface board are conductively engaged with respective, corresponding traces on the lower surface of the membrane assembly extending the co-planar waveguide to the contacts on the membrane. The impedance of the transition signal path from the coaxial cable to the coplanar waveguide on the membrane is, ideally, optimized, with a typical value of 50 ohms (Ω). However, inconsistencies in connections with the ground plane of the interface board may cause the impedance of a particular signal path to vary from the desired matched impedance producing a reflection of the RF signals that are absorbed by other structures resulting in erratic performance of the probing system.
The membrane of the probing system is supported by a support element that is made of an incompressible material, such as a hard polymer, and detachably affixed to the upper surface of the interface board. The support element includes a forward support or plunger portion that protrudes though a central aperture in the interface board to project below the interface board. The forward support has the shape of a truncated pyramid with a flat forward support surface. The membrane assembly which is also detachably secured to the interface board by the support element includes a center portion that extends over and is separated from the forward support surface of the support element by an intervening elastomeric layer. The flexible membrane assembly comprises one or more plies of insulating sheeting, such as polyimide film. Flexible conductive layers or strips are provided between or on these layers to form power/data/signal traces that interconnect with the traces on the interface board at one end. The second end of the traces on the membrane terminate in conductive connections to respective contacts which are arranged on the lower surface of the portion of the membrane extending over the forward support. The contacts are arranged in a pattern suitable for contacting the bond pads of the DUT when the chuck is moved to bring the contacts of the probe assembly into pressing engagement with the bond pads.
The contacts of the probing system comprise a beam which is affixed to the lower surface of the membrane assembly and which is conductively interconnected with the appropriate trace on the surface of the membrane. A contact bump or tip for engaging a bond pad of the DUT is affixed to one end of the beam. When the contact bump is pressed against the bond pad of the DUT, the membrane assembly is deflected, compressing a portion of the elastomeric layer proximate the end of the beam to which the contact bump is affixed. The compliance of the elastomeric layer enables relative displacement of the respective contact bumps and facilitates simultaneous engagement with a plurality of bond pads that may have respective contact surfaces that lie in different planes. The resilience of the elastomeric layer controls the force exerted by the contacts and returns the contacts to the at-rest position when the probe is withdrawn from pressing engagement with the DUT.
The bond pads on DUTs are subject to the rapid development of a layer of oxidation which can electrically insulate the bond pad from the contact. To improve the conductivity of the bond pad/contact interface, the contacts of membrane probes are commonly pressed into the bond pad with sufficient force to penetrate the oxide layer. While penetration of the oxide layer improves conductivity, excessive force can damage the bond pad. With the membrane probe disclosed by Gleason et al, the force of contact with the bond pad is exerted at one end of the beam and the off-center loading on the beam causes the beam to rotate as the portion elastomeric layer adjacent the deflected end of the beam is compressed. The rotation of the beam causes the surface of the contact bump to translate across the bond pad surface and abrade or scrub the oxide coating on the surface improving conductivity between the bond pad and the contact.
However, the conductors within the membrane assembly and attached to the contacts can be broken by excessive displacement of the contacts or may fail from fatigue due to repeated bending when the contacts are displaced during probing. In addition, bond pad material may build up in the area of the contacts of wafer probing assemblies requiring frequent cleaning and, eventual replacement due to wear. While the membrane assembly is detachable from the interface card for cleaning or replacement, the membrane assembly is complex and fairly expensive to replace.
What is desired, therefore, is a probing apparatus having improved impedance characteristics, longer service life and less expensive contacts that can be quickly replaced.
FIG. 1 is a perspective view showing an exemplary membrane probing assembly bolted to a probe head and a wafer supported on a chuck in suitable position for probing by this assembly.
FIG. 2 is a bottom plan view showing various parts of the probing assembly of FIG. 1, including a support element and flexible membrane assembly, and a fragmentary view of a probe card having power, and data/signal lines connected with corresponding lines on the membrane assembly.
FIG. 3 is a side elevation view of the membrane probing assembly of FIG. 1 where a portion of the membrane assembly has been cut away to expose hidden portions of the support element.
FIG. 4 is a schematic sectional elevation view of an exemplary probing assembly.
FIG. 5 is a top plan view of an exemplary support element.
FIG. 6 is a bottom plan view of an exemplary coupon.
FIG. 7 is a sectional view of a membrane and coupon taken along line 7-7 of FIG. 6.
FIG. 8 is a sectional view of a membrane and coupon taken along line 8-8 of FIG. 6.
FIG. 9 is a sectional view of a membrane and coupon illustrating connection to a conductor extending through the forward support.
FIG. 10 is a sectional view of a membrane and coupon illustrating an alternative method of connecting a contact and a membrane supported trace.
FIG. 11 is a sectional view of a membrane and coupon illustrating an alternative method of connecting a contact and a membrane supported trace.
FIG. 12 is a sectional view of a coupon useful with a space transformer of a needle card-type probing apparatus.
FIG. 13 is a sectional view of additional embodiments of contacts for a coupon useful with a membrane probing device.
FIG. 14 is a bottom plan view of a multi-layered tile portion of a coupon.
FIG. 15 is a sectional view of the multi-layered tile of FIG. 14 taken along line 15-15.
FIG. 16 is a sectional view of the multi-layered tile of FIG. 14 taken along line 16-16.
FIG. 17 is a top view of a portion of an exemplary interface board and an impedance optimized co-axial cable to co-planar waveguide interface.
FIG. 18 is a sectional view of the exemplary co-axial cable to co-planar waveguide interface taken along line 18-18 of FIG. 17.
FIG. 19 is a top view of an alternative impedance optimized co-axial cable to co-planar waveguide interface.
Referring in detail to the drawings where similar parts are identified by like reference numerals, and, more particularly to FIGS. 1 and 4, a probe head 40 for mounting a membrane probing assembly 42 is illustrated. In order to measure the electrical performance of the electrical circuit or device-under-test (DUT) of a particular die area 44 included on the silicon wafer 46, input and output ports of the test instrumentation 49 are communicatively connected to contacts 81 included on the lower portion of the membrane probing assembly and the chuck 51 which supports the wafer is moved in mutually perpendicular X, Y, and Z directions in order to bring bond pads of the DUT into pressing engagement with the probe's contacts.
The probe head 40 includes an interface board 52 on which traces 48 and shielded transmission lines are arranged for communicating data, signals and power between the test instrumentation and the DUT. Typically, high frequency signals are communicated between the test instrumentation and the probe with co-axial cables 53 which connect the instrumentation to co-axial cable adapters 55 on the interface board. The shielded transmission lines, typically comprising second lengths of co-axial cable 50 connect the adapters to metallic traces 87 on the interface board to transition the communication path from the coaxial cable to a co-planar waveguide. Referring also to FIG. 2, contact portions 86 that terminate conductors on surface of the membrane are arranged to overlap and conductively interconnect with the traces on the interface board extending the co-planar waveguide from the interface board to the lower surface of the membrane assembly in the vicinity of the contacts and, eventually, to appropriate contacts 81 of the probing apparatus.
Referring also to FIG. 3, the membrane probing assembly 42 includes a support element 54 formed of incompressible material such as a transparent, hard polymer. This element is detachably connected to the upper side of the interface board by means of four Allen screws 56 and corresponding nuts 58 (each screw passes through a respective attachment arm 60 of the support element, and a separate backing element 62 that evenly distributes the clamping pressure of the screws over the entire back side of the supporting element). This detachable connection enables different membrane assemblies having different arrangements of conductors to be quickly substituted for each other as needed for probing different devices. A transparent window 63 enables an operator of the probing apparatus to view the portion of the wafer in the vicinity of the probe contacts 81.
Referring to also FIG. 5, the support element 54 includes a rearward base portion 64 to which the attachment arms 60 are integrally joined. Also included on the support element 54 is a forward support or plunger 66 that projects outwardly from the flat base portion. This forward support has angled sides 68 that converge toward a flat support surface 70 so as to give the forward support the shape of a truncated pyramid. Referring also to FIG. 2, a flexible membrane assembly 72 is attached to the support after being aligned by means of alignment pins 74 included on the base portion. This flexible membrane assembly is formed by one or more plies of insulating sheeting such as KAPTON™ polyimide film sold by E. I. Du Pont de Nemours or other polyimide film, and flexible conductive layers or strips are provided on the surfaces of the plies or between the plies to form various conductors connecting the probe's contacts with the traces and shielded transmission lines on the interface board.
When the support element 54 is mounted on the upper side of the interface board 52 as shown in FIG. 3, the forward support 66 protrudes through a central opening 78 in the interface board so as to present the contacts 81 which are arranged on a coupon 80 that is attached to the surface of the membrane assembly 72 in a position suitable for pressing engagement with the pads of the DUT. Referring to FIG. 2, the membrane assembly includes radially extending arm segments 82 that are separated by inwardly curving edges 84 that give the assembly the shape of a formee cross, and these segments extend in an inclined manner along the angled sides 68 thereby clearing any upright components surrounding the pads. A series of contact pads 86 terminate the conductors of the membrane so that when the support element is mounted, these pads electrically engage corresponding termination pads provided on the upper side of the interface board so that the traces 48 and shielded transmission lines 50 on the interface board are conductively connected to the contacts 81 on the coupon 80.
The exemplary membrane probing assembly 42 is capable of probing a dense arrangement of contact pads over a large number of contact cycles in a manner that ensures reliable electrical connection between the contacts and pads during each cycle despite oxide buildup on the pads. This capability is a function of the construction and interconnection of the support element 54, the flexible membrane assembly 72 and the coupon 80. In particular, the membrane probing assembly is so constructed and connected to the support element to enable the contacts to engage a plurality of bond pads on the DUT even if the contact surfaces of the pads are not co-planar. Moreover, the contacts on the membrane assembly preferably wipe or scrub, in a locally controlled manner, laterally across the pads when brought into pressing engagement with the pads. Alternatively, the contacts may be constructed to enable the tips of the contacts to penetrate an oxide coating on the surfaces of the pads with substantially vertical motion. In the event that the contacts require replacement due to, for examples, a built up of pad material, wear, or a change in the arrangement of the pads to be probed, the contacts can be easily replaced by removing and replacing the coupon without the added expense of replacing the membrane assembly.
Referring also to FIGS. 7 and 8, for ease of illustration, an exemplary flexible membrane assembly 72 (indicated by a bracket) comprises one or more traces 76 affixed to an outer surface of a ply 92 comprising a dielectric material which is separated by a backplane conductive layer 94 from a second dielectric ply 96. However, additional dielectric plies and conductive layers can be used in accordance with conventional layering techniques. Preferably, the conductors of the backplane layer and the other traces are fabricated using a photolithographic process to enable precise control of the dimensions of the conductors. The membrane assembly is secured along its edges and extends over the flat support surface 70 of the plunger 66. Typically, the plurality of traces 76 affixed to the outer surface of the membrane assembly provide communication paths for conducting data/signals and/or power from the conductors at the interface board to the surface of the membrane proximate the forward surface of the plunger. In conjunction with the backplane conductive layer, the conductive traces on the outer surface of the membrane assembly comprise controlled impedance communication structures connecting the interface board to the surface of the membrane in the vicinity of the forward surface of the plunger. The backplane conductive layer 94 may include portions defining one or more apertures which, in turn, define conductive strips of the backplane layer that are located proximate the various traces 76 on the outer surface of the membrane assembly. Filter capacitors 83 may be affixed to the membrane to provide frequency dependent interconnection between the conductive backplane layer and the power and signal conductors.
FIG. 8 shows an enlarged plan view of the lower surface of an exemplary coupon 80 that is attachable to the lower surface of the membrane 72 in area of the forward support surface 70 of the plunger. Typically, the coupon is attached to the surface of the membrane by an adhesive 95 that enables removal of the coupon by breaking the adhesive bond. The adhesive may be applied proximate the periphery of the coupon, at particular locations on the surface or as layer that substantially covers the surface of the coupon but which is displaced from the points of electrical contact by pressure applied to the lower surface of the coupon. The coupon includes a plurality of contacts 81 which in the illustrated exemplary embodiment are arranged in a square pattern suitable for engaging a square-like arrangement of bond pads. Referring also to FIGS. 7 and 8, which represent sectional views taken along lines 7-7 and 8-8, respectively, in FIG. 6, the coupon comprises an elastomeric layer 98 that is substantially co-extensive with the exposed surface of the portion of the membrane that covers the forward support surface 70 of the plunger. The elastomeric layer can be formed by a silicon rubber compound such as ELMER'S STICK-ALL™ silicon rubber made by the Borden Company or Sylgard 182 silicon rubber from Dow Corning Corporation. Each contact comprises a relatively thick rigid elongate beam 100, preferably 150 to 250 microns long, having an upper surface that is affixed to the elastomeric layer. At one end of the beam a contact tip 102 having the general shape of a truncated pyramid is formed. For wear resistance, the contact tip may include a tip end portion 104 comprising an alloy of rhodium and nickel. Proximate the end of the beam opposite of the contact tip and on the opposite side of the beam, a post portion 106 projects in the direction opposite of the contact tip. The post 106 projects through the elastomeric layer and, preferably, projects proud of the upper surface of the coupon. As illustrated in FIG. 7, the posts of selected contacts are arranged for conductive contact with one of the traces 76 on the outer surface of the membrane. As illustrated in FIG. 8, the posts of other contacts are arranged for conductive contact with contact buttons 108 projecting from the backplane conductive layer 94 through the lower ply 92 of the membrane assembly. Additional dielectric layers, for example layer 112, may be affixed to the lower surface of the elastic layer.
As indicated in FIG. 7, when an exemplary contact 102 is brought into pressing engagement with a respective bond pad 110, the contact force is exerted upward at the end of the beam to which the contact tip is affixed. However, the opposite end of the beam is constrained by the contact between the post 106 and the trace which is supported by the substantially incompressible membrane. The resulting off-center force on the rigid beam 100 causes the beam to pivot or tilt against the elastic recovery force exerted on the beam by the elastically supported lower surface of elastomeric layer 98. This tilting motion is localized in the sense that the portion of the beam proximate the tip moves a greater distance toward the forward support surface 70 than a rearward portion of the same beam. The effect is such as to cause the tip of the contact to translate across the pad in a lateral scrubbing movement with a solid-line and a dashed-line representing the beginning 102 and ending positions 102′, respectively, of the contact tip on the pad 110 which, after displacement relative to the coupon, will be in position 110′. In this fashion, the insulating oxide buildup on the pad is abraded so as to provide reliable probe contact-to-bond pad electrical connections. On the other hand, the movement of the contact does not cause flexing of the trace or other conductor of the membrane which substantially reducing fatigue of the conductors and extending the service life of the membrane.
Referring to FIG. 9, the power transmissible on the traces deposited on the plies of the flexible membrane is limited by the cross-section of the traces. As a result, extremely wide traces or large numbers of traces may be required to provide sufficient power to operate the electrical circuits of a DUT or a plurality of DUTs. To increase the power available for testing single or multiple DUTs and reduce the number and size of traces on the membrane, the probing assembly 42 includes a power conductor 114 that is routed through the support 66 to a contact button 116 that extends through the lower wall of the forward support surface 70. The contact button conductively engages a buss 118, which may comprise a grid, On the upper surface of the membrane 72. At selected locations, the buss is extended downward to the lower surface of the membrane through apertures 119 in the other layers of the membrane. Selected contacts 81 include post portions that engage the buss extensions to provide power to the appropriate bond pad(s) of the DUT(s). The buss can have a substantially greater cross-section than the other conductors on the membrane because it is not located in an area of the membrane where flexibility is required.
The resilient elastomeric layer 98 of the coupon, backed by the incompressible support surface 70 and the substantially incompressible membrane 72, exerts a recovery force on each tilting beam contact and thus each contact tip to maintain an adequate level of contact tip-to-bond pad pressure during scrubbing. At the same time, the elastomeric layer accommodates minor height variations between the respective contacts and pads. Thus, when a relatively shorter contact is situated between an immediately adjacent pair of relatively taller contacts and these taller contacts are brought into engagement with their respective pads, deformation of the elastomeric layer enables the shorter contact to be brought into engagement with its pad after only a small amount of further over travel by the longer contact tips. Similarly, the compressibility of the elastomeric layer enables the contact tips to be brought into proper pressing engagement with a plurality of bond pads having surfaces that are not co-planar.
Referring to FIGS. 10 and 11, alternatively, the contacts 120 may comprise an elongate beam portion 122 affixed to the elastically supported lower surface of the elastomer layer 98. The contact tip portion 102 is affixed proximate one end of the beam portion. When the tip is brought into pressing engagement with the bond pad of the DUT, the lower surface of the elastomer layer is displaced proximate the end of the beam to which the tip is affixed, but the elastic force exerted at other end of the beam causes the beam to tilt facilitating the scrubbing action when the tip is pressed toward the bond pad of the DUT. The contact 120 is conductively connected to one of the traces 76 or the backplane layer 94 by a flexible conductive pigtail 124. The conductive pigtail may terminate on the surface of the elastomeric layer at a location suitable for contact with a trace 76 or a contact button 108 or maybe conductively connected with a contact pad 126 suitably located on the surface of the coupon for contact with the appropriate trace or contact button.
FIG. 12 illustrates a coupon 150 (indicated by a bracket) having a plurality of elastomer suspended contacts that is suitable for use with components of a needle card-type probe head enabling the needle card-type probing assembly to be converted to a lower inductance probing assembly with elastically suspended contacts by removing the needle card-type probe head and replacing it with the coupon 150 that interfaces with a space transformer 152 (indicated with a bracket) of the needle card-type probing apparatus. In the schematic cross-sectional view of FIG. 12, the size of certain elements and components are exaggerated, for illustrative clarity. The interface board 154 of the needle card-type probing assembly is generally a conventional circuit board substrate having a plurality of terminals 156 (two of many shown) disposed on a surface thereof. The terminals provide an interface for wires 158 that connect instrumentation (not shown) to the probing assembly. As illustrated, the wires may be connected to terminals on one side of the interface board which are, in turn, connected by conductive vias 160 to terminals 162 or traces on the opposing side of the interface board. Additional components (not shown), such as active and passive electronic components, connectors, and the like, may be mounted on the interface board and connected to additional terminals. The interface board is typically round and commonly has a diameter on the order of 12 inches. The terminals 156, 162 on the interface board are often arranged at a 100 mil pitch or separation distance.
While some probing assemblies do not utilize an interposer, the exemplary probing assembly of FIG. 12 includes an interposer 164 (indicated by a bracket) disposed between the interface board and the space transformer. An interposer comprises interconnected electrical contacts disposed on opposing sides of a substrate so that components on opposing sides of the substrate can be conductively interconnected. An interposer is often used in a probing assembly to facilitate reliable conductive connection between the terminals of an interface board and the terminals of a space transformer. The interposer also aids in accommodating differences in thermal expansion of the interface board and the space transformer. The interposer 164 comprises a substrate 166 and a plurality of fuzz buttons 168 (two are shown) that protrude through holes in the substrate. The fuzz buttons each comprise a fine wire that is compressed into a small cylindrical shape to produce an electrically conductive, elastic wire mass. As a general proposition, the fuzz buttons are arranged at a pitch which matches that of the terminals 162 of the interface board. One end of each of the conductive fuzz buttons is in contact with a terminal on the interface board while the second end of the fuzz button is in contact with a terminal 170 on the space transformer. The elastic fuzz buttons are compressed providing compliance to accommodate variations in the separation distances between of the various terminals of the interface board and the space transformer and exerting pressure on the contacts to promote good conductivity.
The fuzz buttons protruding through the substrate of the interposer 164 contact conductive terminals 170 on one side of the space transformer 152. The space transformer 154 (indicated by a bracket) comprises a suitably circuited substrate 172, such as a multi-layer ceramic substrate having a plurality of terminals (contact areas, pads) 170 (two of many shown) disposed on the surface adjacent to the interposer and a plurality of terminals 174 (contact areas, pads) (two of many shown) disposed on the opposing surface. In the exemplary probing assembly, the contact pads adjacent the interposer are disposed at the pitch of the terminals of the interface board, and the contact pads 174 arranged on the opposing surface of the space transformer are disposed at a finer pitch corresponding to the pitch and arrangement of the needle-type probes included in the needle card with which the space transformer was intended to interface. While the pitch of the terminals of the interface board is commonly approximately 100 mil, the pitch of needle-type probes can be as fine as approximately 125 μm. Conductive traces 176 in the multilayer substrate of the space transformer re-route the electrical connections from the finely pitched pattern required to interface with the probe head to the more coarsely pitched pattern that is obtainable with a printed circuit board, such as the interface board.
The various elements of the probing assembly are stacked and any suitable mechanism for stacking these components and ensuring reliable electrical contacts may be employed. As illustrated, the probing assembly includes a rigid front mounting plate 180 arranged on one side of the interface board. A stand-off 182 with a central aperture to receive the space transformer is attached to the front mounting plate. A mounting ring 184 which is preferably made of a springy material such as phosphor bronze and which may have a pattern of springy tabs extending therefrom, is attachable by screws 186 to the stand-off with the space transformer captured between the mounting ring and the stand-off.
A coupon 150 (indicated by a bracket) comprising an elastomeric layer 190 and a plurality of electrically conductive contacts is affixed to the face of the space transformer, preferably with an adhesive 191. The contacts, for example exemplary contact 192, may comprise, generally, a relatively thick, elongate, rigid beam portion 194 with a post portion 196 proximate one end of the beam and a contact tip 102 projecting in the opposite direction from the opposite side of the beam proximate the second end of the beam. Although other shapes and materials may be utilized, the contact tip preferably has the general shape of a truncated pyramid and the distal end of the contact tip may be coated with a layer of nickel and/or rhodium to provide good electrical conductivity and wear resistant when the contact tip is repeatedly pressed into engagement with the bond pads of DUTs. The post 196 has a rounded end distal of the beam that abuts a terminal 174 of the space transformer 152. The rounded end facilitates movable contact between the post and the terminal when the contact tip is displaced upward by interaction with a bond pad. Additional dielectric layer(s) 198 may be affixed to the lower surface of the elastomeric layer.
The contact 200 exemplifies a second type of contact that may included in the coupon. The contact 200 comprises a contact tip portion 202 which, preferably, has the shape of a truncated pyramid or cone and a body 204 with, preferably, a square or circular cross-section. A shoulder 206 may project from the body adjacent to the bottom surface of the elastomeric layer. The contact is conductively connected to the upper surface of the coupon 150 by a pig-tail 204 that is conductively attached to the body of the contact at one end and has a second end exposed at the upper surface of the coupon. The exposed portion of the pig-tail is arranged to contact a terminal 174 of the space transformer when the coupon is affixed to the surface of the transformer. Alternatively, a conductive connection between the contact and the terminal of the space transformer may incorporate a fuzz button, similar to fuzz button 168, that is embedded in the elastomeric layer with one end exposed at the surface of the coupon and the second end abutting the contact.
For better conductivity between a bond pad and the contact, the tip 202 of the contact 200 is intended to be pushed through the oxide layer that may develop on the surface of a bond pad and the elasticity of the elastomeric layer may be varied to aid penetration of the oxide layer. For example, the body 204 of the contact may be extended so as to directly contact the terminal of the space transformer, eliminating the need for the pigtail and enabling vertical movement of the contact in response to pressure from the pad to be limited to the deflection of the relatively rigid space transformer while the resiliency of the elastomeric layer may enable the contact to tilt if the surface of the pad is not parallel to the end surface of the tip of the contact. On the other hand, the elastomeric layer 190 may comprise a single layer with graduated resiliency or a plurality of sub-layers with differing resiliency to enable controlled vertical movement of the contact in response to the application of force at the tip of the contact. The coupon provides an economical way of converting a needle-type probing apparatus to a probing apparatus with much lower inductance. The relatively long, closely spaced, needle-like probes typically exhibit a single path inductance of 1-2 nano-Henrys (nH) which is sufficient to substantially distort high frequency signals and limit the usefulness of needle-type probes for testing devices with high frequency signals. On the other hand, single path inductance of 0.2 nH has been demonstrated with elastically suspended probes of membrane-type probing apparatuses.
Referring to FIG. 13, surface penetrating contacts 252, 254 can also be utilized in a coupon 250 (indicated by a bracket) suitable for use with a membrane probing apparatus. The contact 254 may be rigidly supported by the support 66 and the substantially incompressible membrane assembly 72. As illustrated, the body of the contact is extended through the elastomeric layer 98 so that it abuts the contact button 108 of the backplane layer 94. The abutting end of the contact is rounded enabling tilting of the contact, by compressing the elastomer surrounding the contact's body, to aid in aligning the end of the contact with a non-co-planar pad surface.
On the other hand, the body of the contact 252 terminates in the elastomeric layer and the contact can exhibit elastic behavior as determined by the properties of the elastomer layer. The elastomeric layer 98 may comprise sub-layers 98A and 98B, each having different elastic properties. The elasticity of the contact during vertical displacement may be determined by the properties of the sub-layer 98A which acts on a portion of the surface area of the upper end of the contact and the properties of sub-layer 98B which acts on the upper surface of a contact flange 253. A fuzz button 256 conductively connects the contact 252 with the trace 76.
Referring to FIGS. 14 and 15, a coupon 300 for use with a membrane probe or a space transformer may comprise a plurality of coupons or tiles 300a, 300b, 300c, 300d, each independently detachably affixed to a membrane or a space transformer enabling damaged or worn portions of the coupon to be independently replaced or portions of the coupon to include differing arrangements of contacts 81. Flexibility in arranging contacts and conductive connections to membranes and space transformers can be facilitated by layering the conductive and non-conductive elements of the coupon. For example, the tile 300c includes layered circuitry enabling conductive connection of thirteen finely pitched contact tips to a space transformer or membrane having six coarsely pitched contact pads enabling a typically less costly, more coarsely pitched space transformer or membrane to interface with the finely pitched contacts useful for probing small DUTs. The tile includes six grounded contact tips 302 including ground contact tips spaced to each side of a pair of high frequency signal contact tips 304 comprising the terminations of the conductors of a pair of ground-signal-ground (GSG) co-planar waveguides 310 (collectively indicated by a bracket). In addition, the exemplary tile includes a pair of contact tips 308 that are arranged to conduct DC power to a DUT and a pair of contact tips 306 that are arranged to conduct a lower frequency signal to the DUT. The contacts tips of the coupon are affixed to an elastically supported lower surface 314 of an elastomer layer 312. The coupon is affixed the lower surface to a space transformer 320 by adhesive 322.
The ground contact tips 302 are conductively connected to a conductive backplane layer 318 deposited between a pair of dielectric layers 320 and 321. The elastomeric layer is affixed to lower surface of the dielectric layer 321. The backplane layer extends over the area of the coupon that is occupied by the contact tips 302-308. The backplane layer is conductively connected to a suitable conductor in the space transformer by a contact button 324 on the upper surface of the coupon which is connected to the backplane layer by a via 326. The contact button 324 is arranged to engage a corresponding contact button on the lower surface of the space transformer. The contact tips 302 include post portions 327 in contact with backplane or a pigtail 328 that conductively connects the side of the beam portion of the contact tip and the back plane layer.
High frequency signals are transmitted between the test instrument and the DUT through the high frequency contact tips 304. Each of the high frequency contact tips is connected to one of a pair of contact buttons 330 on the upper surface of the coupon by a via 332 and a high frequency signal trace 324 that is deposited between the upper surface of the elastomeric layer 312 and the dielectric layer 321 which separates the high frequency trace from the backplane layer.
The contact tips 308 facilitating the transmission of DC power to the DUT are connected to an appropriate conductor 334 in the space transformer through a contact button 336 on the upper surface of the coupon. A trace 338, including portions that are conductively affixed to the lower surfaces of the beam portions of the power contact tips 308, is affixed to the lower surface of a dielectric layer 340 that is, in turn, affixed to the lower surface of the elastomer layer. A via 342 connects the power trace on the lower surface of the coupon to a contact button 344 on the upper surface of the coupon.
The coupon 300c also enables communication of a lower frequency AC signal to the DUT through contact tips 306 which include posts conductively engaging a lower frequency signal layer 346. The lower frequency signal is conducted from the lower frequency signal layer to the space transformer through a via 348 connecting the lower frequency signal layer with a low frequency signal contact button 350 on the upper surface of the coupon. A multilayered coupon such as the coupon 300 including the tile 300c provides flexibility for connecting an arrangement of finely pitched contact tips with the more coarsely pitched contacts of more moderately priced space transformers.
Referring to FIG. 16, the coupon 300 also includes one or more forming posts 370 that extend from the upper surface of the elastomer layer 312 to the lower surface of the elastomer layer. The forming posts aid in controlling the thickness and in maintaining the planarity of the surfaces of the elastomeric layer during the curing of the elastomer but are not conductively connected to a circuit of the probing apparatus. Posts 372, extending proud of the surfaces of the coupon are not arranged to conduct electricity but may be provided as an aid for the user in gauging the deflection of the elastically suspended contacts. For example, a post 372 having a first end in contact with the space transformer and a second end projecting from the bottom of the coupon a distance slightly less than the height of a contact tip may serve as a contact gage to prevent damage to the coupon from excess deflection of the elastically supported contact tips. Similarly, an elastically supported post 374 that projects below the lower surface of the coupon may be arranged to provide a signal of the limits of contact tip deflection. When the elastically suspended contact tip is pushed upward by engagement with the wafer, the end of the post contacts and conductively connects traces 376, 378 in the coupon to close an electrical circuit and provide a signal at a contact button 380 which may be connected to an indicator circuit.
At RF frequencies, signal frequencies in the radio frequency and microwave frequency ranges, it is important to match the impedance, typically 50 Ω, of the coaxial cable that typically interconnects the test instrumentation and the probing apparatus with the impedance of the coplanar waveguide that extends from the interface board 52 of the probing apparatus to the lower surface of the membrane in the area of the forward support surface 70. A mismatch of impedance will produce reflections of the signals which can couple to adjacent structures causing frequency dependent inaccuracies in the measurements of the DUT's performance. Referring to FIGS. 4 and 17, to transition the signal path from a coaxial cable 50 to the co-planar waveguide comprising traces on the membrane, the conductors of the coaxial cable are first conductively connected to respective traces 87 on the surface of the interface board which collectively comprise an interface board co-planar waveguide. Typically, the center conductor of the coaxial cable is connected to a center trace and the outer conductor of the cable, which is connected to a ground potential, is connected a pair of side traces, one spaced to either side of the center trace to form a ground-signal-ground (GSG) co-planar waveguide. The traces of the interface board co-planar waveguide terminate proximate the edge of the central opening 78 in the interface board and portions of the conductive traces on the membrane's lower surface are arranged to overlap portions of the surfaces of respective traces of the interface board co-planar waveguide when the membrane is clamped to the interface board. The overlapping surface portions conductively interconnect the traces of interface board waveguide to respective traces of a membrane waveguide extending the signal path to the lower surface of the membrane in the region proximate the forward support surface 70.
Typically, the interface board includes a sub-surface conductive layer that is connected to a ground potential to provide a ground plane for the interface board co-planar waveguide. However, inconsistency in the interconnection of the interface board co-planar waveguide and the ground plane may produce an impedance mismatch as the signal path transitions from coaxial cable to co-planar waveguide. The inventors realized that eliminating the interconnections with the interface board ground plane would eliminate this source of error but would also increase the impedance of the interface board co-planar waveguide to unacceptable levels. The inventors came to the unexpected conclusion that the interface board ground plane could be eliminated by decreasing the gaps between the signal trace and the ground traces of the coplanar waveguide, interconnecting the ground traces of adjacent waveguides and providing a low impedance structure for interconnecting the traces of the membrane and interface board waveguides.
The exemplary interface board 500 includes a plurality of conductive traces deposited on the upper surface. Three traces 502, 504, 506 comprise an interface board co-planar waveguide 508 (indicated by a bracket). The center conductor 510 of a coaxial cable 502 is conductively interconnected with the center trace 502 of the waveguide and the outer conductor 514 of the cable is connected to a first side trace 504 spaced apart from one edge of the center trace and a second side trace 506 spaced apart from the second edge of the center trace. Typically, a signal is communicated between the test instrumentation and the probing apparatus over the center conductor of the coaxial cable and the outer conductor is connected to a ground level electrical potential. The interface board ground plane 516, if any, terminates a substantial distance from the area of the interface board that is overlaid by the traces of the waveguide and is not electrically interconnected with the traces of the waveguide. To reduce the impedance of the "ungrounded" waveguide, the width of the center trace 502 differs from the widths of the side traces and the widths of the respective side traces may also differ from each other. For example, the center trace or signal conductor of an exemplary GSG waveguide having an impedance of 50 Ω is 0.0179 inches wide and is separated from each of the spaced apart side traces of the waveguide by a gap of 0.004 inches. If a side trace, for example the side trace 506 of the exemplary waveguide is adjacent to the center traces of two waveguides, for example center trace 502 and the center trace 522 of a second waveguide 520 (indicated by a bracket), the side trace is shared by the adjacent waveguides 508, 512. The side trace preferably has a width of 0.029 inches and the outer conductors 514 of two adjacent coaxial cables 512, 526 are conductively connected to the trace. If a side trace, for example the side trace 504 of the exemplary interface board co-planar waveguide 508, is adjacent to only one center trace, the side trace has a width of 0.0107 inches.
Referring to also FIG. 19, while the impedance of a co-planar waveguide is substantially affected by the width of the gap between the traces of the waveguide, to improve the tolerance of the waveguide to misalignment between overlapping portions of the traces of the membrane waveguide and the traces of the interface board waveguide, the width of the center trace of the interface board waveguide may be narrowed for the portion of the length that is overlapped by the membrane trace. For example, the exemplary waveguide 600 includes a central trace 602 comprising a first length portion 604 that is preferably 0.0179 inches wide and a second length portion 606 which is overlapped by the membrane trace 614 and which is preferably 0.0159 inches wide. The gap 608 between the center trace and the side traces for the first length portion is 0.004 inches while the gap 610 for the second length portion is 0.005 inches increasing the tolerance to misalignment between the traces of interface and membrane waveguides. However, the impedance of the exemplary waveguide 600 is substantially the same as the impedance of a waveguide with uniform 0.004 inch gaps between traces.
The interconnection of the respective traces of the interface board waveguide and the corresponding traces of the membrane waveguide substantially effects the impedance of the waveguide. The electromagnetic field is carried in the gap between the traces of a waveguide and the interconnection between the traces can produce a "pinch point" that substantially increases the waveguide's impedance. The inventors concluded that the more closely the geometry of the interconnecting structure resembles the geometry of the waveguide's traces the lower the impedance of the connection. Referring also to FIG. 18, when the membrane probe is assembled, contact portions 86 of the traces on the membrane overlap portions of the surfaces of the traces on the interface board. The inventors concluded that a signal path comprising a plurality of raised pads 540 on one of the overlapping surfaces of a pair of traces would facilitate a low impedance interconnection between the interface board waveguide and the membrane waveguide. The interconnecting structure for the center traces 502 and 530 of the interface board and membrane waveguides, respectively, comprises four pads 540 that project from the surface of one the traces into contact with the surface of the other trace. A pair of the raised pads, indicated generally as 542 and 544, is arranged substantially parallel to each of the edges of the center trace on which the pads are affixed and the pads are located as close to the edge of the trace as tolerance for misalignment of the trace will allow. Preferably, a raised pad is located at each of the corners of a substantially rectangular area on the surface of the trace.
Similarly, a portion of the side trace 534 on the membrane overlaps a portion of the side trace 504 on the surface of the interface board 500 and a portion of the membrane side trace 532 overlaps a portion of the interface board waveguide side trace 506. On the side (ground) traces, the pair 546 of raised pads 540 nearest the center trace 502 are located as close to the inner edge of the trace as is permitted by the tolerance to misalignment and the second pair 548 of raised pads are preferably located nearer the center line of the membrane trace than the second edge of the trace. The electromagnetic field is carried in the region between the traces of the waveguide and locating the pads of interconnecting structure closer to the region in which the electro-magnetic field is confined reduces the inductance of the connection.
The accuracy of measurements performed with the membrane probing apparatus, particularly at frequencies in the radio and microwave frequency ranges, is improved by optimizing the impedance of the interface between co-axial cables, interconnecting the test instrumentation and the probing apparatus, and the co-planar waveguide that extends the signal path to the contact tips. A longer service life and lower operating cost for a membrane probing apparatus is achievable by including the contact tips on an elastic coupon that is attachable to a membrane that includes a plurality of traces over which signals, data and power can be transmitted. Replacing the probe needles of a needle type probing apparatus with a coupon attachable to a space transformer and including elastically supported contact tips can also improve the performance of a needle-type probing apparatus by substantially lower the inductance of the apparatus.
The detailed description, above, sets forth numerous specific details to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuitry have not been described in detail to avoid obscuring the present invention.
All the references cited herein are incorporated by reference.
The terms and expressions that have been employed in the foregoing specification are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims that follow.


text:
Right now, only the OMP_TARGET_OFFLOAD=DISABLED was implemented. Added support for the other MANDATORY and DEFAULT values.
- Reviewers
gtbercea ABataev grokos caomhin Hahnfeld
- Commits
- rOMP340542: [OpenMP][libomptarget] Bringing up to spec with respect to OMP_TARGET_OFFLOAD…
rL340542: [OpenMP][libomptarget] Bringing up to spec with respect to OMP_TARGET_OFFLOAD…
rG1b4a666ba584: [OpenMP][libomptarget] Bringing up to spec with respect to OMP_TARGET_OFFLOAD…
Diff Detail
Event Timeline
Does this patch supersede D44522?
As discussed in there I don't see that DEFAULT means MANDATORY only iff there was a successful offload. From TR7, page 610, lines 17/18:
The DEFAULT value specifies that when one or more target devices are available, the runtime
behaves as if this environment variable is set to MANDATORY [...]
Another remark from D44522: I think we need to handle API methods as well.
There was an agreement with Intel to have a query function __kmpc_get_target_offload. Please use that one.
As discussed in there I don't see that DEFAULT means MANDATORY only iff there was a successful offload. From TR7, page 610, lines 17/18:
The DEFAULT value specifies that when one or more target devices are available, the runtime
behaves as if this environment variable is set to MANDATORY [...]
The question is how we define "available." IMO, it means success of first command. Happy to reconsider, I think most of our users are fine with success of first command
Another remark from D44522: I think we need to handle API methods as well.
I am all for using the suggested __kmpc_get_target_offload function, which is implemented. If we do so, we will be depending on a definition in kmp.h. Do we want a redundant definition of the
enum kmp_target_offload_kind {
tgt_disabled = 0, tgt_default = 1, tgt_mandatory = 2
I assume we want to do that, as we don't want to include kmp.h in libomptarget.
You are right, "available" is not defined in the standard. I've always though of "plugged into the system", ie all devices that are visible to the CUDA runtime. That would match the current implementation of omp_get_num_devices which is defined to return "the number of available target devices".
Actually this behaviour would be important for us as we have our GPUs configured exclusively. So when there is already a process running all other users get a runtime error. In that case it would be very helpful to have libomptarget abort the program.
Another remark from D44522: I think we need to handle API methods as well.
I am all for using the suggested __kmpc_get_target_offload function, which is implemented. If we do so, we will be depending on a definition in kmp.h. Do we want a redundant definition of theenum kmp_target_offload_kind { tgt_disabled = 0, tgt_default = 1, tgt_mandatory = 2 };
I assume we want to do that, as we don't want to include kmp.h in libomptarget.
Yes, D44522 also had that.
Why not move the check of handle_target_outcome into CheckDeviceAndCtors so you dont have to do every time you invoke this function.
So you like deciding available on first use? This is what your comment seems to imply, but I am not 100% sure.
I think that our current interpretation of "available" for devices is
reasonable. There may be many reasons that a device is not available, even
if it is plugged in. Deciding that it is available because we were able to
use it seems the most dynamic method of determining this.
Kevin O'Brien
No, I favor the implication "visible" -> "available" which is the same interpretation that omp_get_num_devices is using (in its current form).
If we implemented the behaviour of this patch ("successful offload to ONE device" -> "ALL devices available", "error on ONE device" -> "NO devices available") we'd need to change the API methods. That would probably imply probing all devices at runtime startup - because after all we don't know which device the user is going to use. IIRC that was to be avoided, libomptarget uses lazy initialization at the moment.
Suppose we have 2 devices plugged into the system, and the first one cannot be used (for whatever reason: hardware failure, exclusive configuration and somebody else is running, etc.).
Now a clever application sees the two (because omp_get_num_devices() returns 2) and does:
#pragma omp parallel num_threads(omp_get_num_devices()) { #pragma omp target device(omp_get_thread_num()) { } }
I think the runtime behaviour with this patch depends on the execution order (and exposes a race condition in handle_target_outcome on TargetOffloadPolicy; let's ignore that for now):
- If target device(0) executes first, libomptarget will notice the error and silently disable offloading. All target regions will execute on the host.
- If however target device(1) executes first and returns successfully, libomptarget will raise OMP_TARGET_OFFLOAD to MANDATORY and will abort execution when catching the error of target device(0).
I don't think that makes much sense. IMO the runtime should detect two "visible" -> "available" devices and abort execution in all cases.
As I understand Jonas, he would prefer semantically something like:
devices-available = omp_get_num_devices()>0
This does not depend on a successful offload, just compares whether a device is there:
if devices-available && DEFAULT: continue as if MANDATORY else: continue as if DISABLED
Yes, because that's how the standard defines omp_get_num_devices(): to return the number of available devices.
And because it results in a sane behaviour for my example.
This does not depend on a successful offload, just compares whether a device is there:if devices-available && DEFAULT: continue as if MANDATORY else: continue as if DISABLED
Almost, !devices-available && MANDATORY should not result in DISABLED ;-)
if DEFAULT: if devices-available: continue as if MANDATORY else: continue as if DISABLED endif
I disagree. As it is currently written, omp_get_num_devices() can also grow over time, so you can also have it return zero, only to increase later to a larger number. What does that do to DEFAULT?
I believe the current policy is ok; if any device fails for any reason on first use, it becomes disabled. Anyone that want to rely on devices being there should use the MANDATORY policy.
I am happy to add a lock around the change of DEFAULT to MANDATORY or DISABLED.
Ok, fair point. I think we need to decide on the first entry from user code: If one construct fell back to the host all following constructs should, shouldn't they?
I believe the current policy is ok; if any device fails for any reason on first use, it becomes disabled. Anyone that want to rely on devices being there should use the MANDATORY policy.
Huh, but there is only one global TargetOffloadPolicy. So how can we disable it = a single device?
Did you consider this example?
Did you consider this example?
Your example can show a different issue as well:
A program first do a register_lib that has code for a device which does not exist on that machine. Thus omp_get_num_devices is zero. DEFAULT then becomes DISABLED. A program then does a register_lib that has code for a device that exist on this machine. But now it is disabled.
A way out of this is to see the bigger picture. If the user want to guarantee execution of all targets on a device, the user must us MANDATORY. If the user does not want devices, then DISABLED is called for. DEFAULT is a best effort, it will not be perfect, nor it has to be perfect.
I suggest that deciding on the first attempt to actually offload is as valid a policy as any, and is simple to understand. Regardless of how we do it, what we want to really avoid is that we execute some kernels on the device, and some on the host, (ignoring here explicit orders from the program via the "if(0)" clause). Both policies "num_devices>0" and "decide on first invocation" satisfy this.
Added a mutex around changing DEFAULT to MANDATORY or DISABLED
Added proper action when discovering failure (which is immediate abort of function being performed)
Failure to locate data during an update is now tolerated (a warning could be issued)
Ravi, I did your change but reverted it for the following reason. I liked to have all of the handling of target_offload variable in the same file (interface.cpp). When I moved in into the CheckDeviceAndCtors, I needed to insert it in 2 places within the function, and handle_target_outcome was now in two different files.
If you feel strongly about your request, I will be happy to move the code into CheckDeviceAndCtors.
Alex: If you feel strongly about your request, I will be happy to move the code into CheckDeviceAndCtors.
It is just a suggestion. Either way is fine.
@Hahnfeld and I discussed the behavior and found a very inconsistent behavior. Let's consider something like the following code, under the constraint, that device(1) is busy (will fail to offload) and device(0) is free:
We found the following cases:
- first offload to device(0) -> succeed -> set MANDATORY -> abort on offloading to device(1)
- first offload to device(1) -> fail -> set DISABLED -> continue execution on the host
- start offloading to device(0) and device(1) -> fail on 1 -> set DISABLED, but execution of offloaded code will complete. -> although we are in DISABLED, we executed code on a device
The locking only prevents the race on the variable, but not the behavior in 3).
From our point of view, the decision between DISABLED and MANDATORY must be before starting any offloading.
Another reason to implement such behavior:
If the offloaded code changes the memory state (and the target region code is not idempotent), this prevents inconsistent states of execution, if the initial offloading fails after partial execution on the device and a second execution the host.
Default policy is now selected when registering the libraries, and will default to mandatory/disabled depending of whether there are one or more devices/none.
Good arguments, I see why the consistency of why linking the number of devices returned and OMP_TARGET_OFFLOAD environment is good.
Implemented the requested changes
|188–194 ↗
|(On Diff #160413)
Please remove, there is a CMake flag to do this.
|368–370 ↗
|(On Diff #160413)
I think this implementation can live directly in interface.cpp as no other file should use it. Please make the function static and I think internal function names follow a CamelCase naming convention; device_is_ready seems to be the exception...
|385–386 ↗
|(On Diff #160413)
Asserts are no-ops when building with -DNDEBUG which is the default for Release builds. Call exit(1) directly?
I think we should provide the user with an error message when aborting the execution, DP is subject to LIBOMPTARGET_DEBUG and only available in Debug builds.
(PGI's OpenACC implementation prints a detailed error code from CUDA which is probably not possible in the target agnostic part. I think that's ok for now, the user can use cuda-gdb or other tools...)
|24–31 ↗
|(On Diff #160413)
Is there a particular reason the code is put into device.h / device.cpp? IMO it's not related to device management.
For later reuse in API methods (if deemed necessary after the discussion on omp-lang) I think this should go into private.h.
I think this code path is not triggered when there is no matching RTL at all. At the moment this causes an assert (in Debug builds) because TargetOffloadPolicy is still tgt_default.
However I don't think we can decide on a single call to __tgt_register_lib for the reason you mentioned last week: The number of available devices can change when more device images are registered. As such I think we should move the decision handling tgt_default to the beginning of all interface methods that can come from a user's target construct. What do you think?
Please also clang-format your changes.
This seems overly complex where we now only need a single fprintf (which is thread-safe) inside an if. Do you have upcoming patches that will use these functions?
Please rename to CamelCase, something like HandleTargetOutcome? (device_is_ready seems to be the exception)
After some thinking I guess this actually works in all cases:
However I don't find this very intuitive and it will be very easy to mess up with future changes. I suggest to do this upfront, for example with a helper function:
static bool IsOffloadDisabled() { if (TargetOffloadPolicy == tgt_default) { // ... } return TargetOffloadPolicy == tgt_disabled; }
The first line of each interface function would then become if (IsOffloadDisabled()) return;. In addition the duplicate code could be dropped from RTLsTy::LoadRTLs. Do you have concerns about this straight-forward solution?
responded to comments
Yes, I intend this to be used for all output feedback to users. Ideally, I would find a way to share the infrastructure in kmpc, but for the moment this should do it. Note that the lock is only acquired prior to aborting, so overheads are not an issue.
good suggestions, re-inserted the lock as now there could be a race condition. Again, in practice, this should not happen often. In practice, the lock should be acquired only once, so this is ok
Looks mostly good.
I'm not sure whether you did this; especially the new functions in interface.cpp don't seem to follow the style I'm used to...
In that case I think the functions should be moved to omptarget.cpp (or a completely new file?) because they'll be used by more than just the interface?
Inverting cond seems weird, could you explain that choice? If that's to model assert I think the function should be renamed to AssertOrFatalMassage. To me the current function name implies if (cond) FatalMessage(...), but that may be just me.
I agree.. maybe private.cpp?
I named them AssertWithFatalMessage, but then you mentioned that asserts are turned off in the release mode, so I though that would be confusing too.
But if you prefer Assert, that was my first choice too
Another option would be to remove FatalMessageWithCond and put in the if statement directly. That would a) be clear, b) avoid choosing a name and whether to invert cond or not, and c) deduplicate code and make FatalMessage used (which is not at the moment).
For one single function I don't think we need to start a new file right now. IMO that could go to omptarget.cpp.
Thanks to all for your valuable comments, much appreciated, really contributed to the quality of the patch
Looking thourhg the code one more time, I realized that there was no default init for this key variable
kmp_target_offload_kind_t TargetOffloadPolicy = tgt_default;
It's better for it to be init to zero value, to be absolutely safe.
Alex, will you land this anytime soon? (I'd like to backport for our local installation of Clang 7.0...)
Anyone else having trouble building after this change?
.../llvm/projects/openmp/libomptarget/src/omptarget.cpp:37:21: error: 'va_start' was not declared in this scope
va_start(args, fmt); ^
.../llvm/projects/openmp/libomptarget/src/omptarget.cpp:43:14: error: 'va_end' was not declared in this scope
va_end(args); ^
It seems my environment would really like a stdarg.h in this file.

subset: commoncrawl


text:
2020 Buyers Guide
Sponsored by:
das Manufacturing, Inc.
Valrico, FL 33596
Since 1983, das Manufacturing Inc. has been offering innovative marking solutions for a wide variety of applications. We are the leader in permanent ground-level surface marking.
das Trail Markers™ can be permanently applied to concrete, asphalt or virtually any flat non-soil surface with das Adhesive. The markers are an unobtrusive yet highly visible method of placing information directly on a trail or path. Brand your trails, mark features, distances and emergency location numbers. Durable, fade resistant, low profile and ADA compliant, they can be used as an alternative to vertical signage clutter.
Stock trail markers are available in a number of shapes and legends. Custom markers can be manufactured in just about any size, shape or color with variable data, distances or QR codes.
Our company also offers:
das Trail Decals™ which can be used for temporary marking or branding for events where removability is necessary.
Doo Dots™ to encourage the proper disposal of pet waste by placing the message at ground level where it will do the most good.
das Blazes™ for marking trees. They are easy to see and easy to customize.
A variety of quality custom signage is also available.
Contact das Manufacturing, Inc. for creative, cost-effective solutions to all of your marking needs.
Categories Listed In:
Building & Facility Components
– Security
– – Signs
– Signage, Interior
Site Furnishings, Structures & Park Equipment Components
– Playground Equipment
– – Safety Signage
– Signs
– – Directional
– – Permanent
Search the Buyers Guide


text:
Schedule 4C pool
The trustees are treated as having a Schedule 4C pool if they make a transfer of value to which TCGA92/Sch4B applies. The pool is made up of three possible elements:
- any outstanding section 2(2) amounts as at the end of the year of transfer, CG39265
- any Schedule 4B gain created by the transfer of value, CG39270
- any Schedule 4B gains created by further transfers of value in the same tax year.
There is no requirement that the transfer of value creates a Schedule 4B gain or that the Schedule 4B gain is going to be taxed on the beneficiaries who receive capital payments. Schedule 4B will still apply to a transfer of value even if the only trust property is non-chargeable assets. Also it is possible that any Schedule 4B gain is taxed on the settlor under TCGA92/S86 leaving no TCGA92/Sch4B to be taxed beneficiaries who receive capital payments.
The significance of this is that it could be that the only amounts in the Schedule 4C pool are the outstanding section 2(2) amounts. This matters because, unlike TCGA92/S87, capital payments to beneficiaries who do not pay UK tax are not matched against the Schedule 4C pool. That means those gains remain available for matching when capital payments are received by UK resident beneficiaries.
It is possible that further amounts could be added to the pool after the year of transfer if the trustees make further transfers of value in later years, CG39275.
If the settlement ceases to exist after the trustees have made a transfer of value the rules in TCGA92/Sch4C apply as if a tax year ended immediately before the cessation, TCGA92/Sch4C/para13A.


text:
Each audition will begin with a short solo which you choose. This portion of the audition will last around two minutes, so be sure to play the part of the solo which puts your best foot forward. Solos may come from Suzuki books, etudes, concertos, solo repertoire, or any other source. Be sure to consult with your private music teacher or school music teacher for help selecting an appropriate solo. Practice, practice, practice!
Play your solo for others and get feedback. Video record yourself and listen with a critical ear. Look at your posture, body language, and technique.
At the audition, bring two photocopies of the solo for the judges. Before you begin, state the title, composer, and movement you will be performing. Play confidently and have fun! Everyone there, including the judges, want you to play well.
Note: Percussion should solo on either Xylophone, Marimba, Snare Drum or Timpani.


text:
Gypsy Star Mystic Jasper Arrowhead on Turquoise necklace by Sage
Regular price $ 235.00
Gypsy Star Mystic Jasper Arrowhead on Turquoise necklace by Sage
"We are all wanderers on this Earth. Our hearts are filled of wonder, and our souls are deep with dreams."
Illuminate your Gypsy Soul with Sage's newest collection, Gypsy Star
*Limited edition, exclusively available at The Sage Lifestyle.
*Made by Sage Machado.
*Proudly made in Los Angeles, California, USA.
*Gold material on pendant and chain, 24K Vermeil.
Necklace length: 18"
Pendant length: 1" W: 1/2
Need help choosing your next signature jewelry piece? Let Sage help guide you!
Phone: (323) 931-0595
Email: email@example.com


text:
Manchester City are to rival Atletico Madrid for the signature of 18-year-old San Lorenzo striker Angel Correa, according to the Mirror.
The wonder kid has been labelled as the next Sergio Aguero and is emerging as one of the brightest young talents in South America.
His performances have caught the attention of some of Europe's biggest clubs, with the likes of Barcelona, Napoli and Arsenal all sending scouts to Argentina to watch him.
But Atletico Madrid have reportedly made the first move, with an official bid placed for Correa, and City are expected to be close behind with a more attractive offer.
San Lorenzo are not keen on selling the striker and want to keep him for as long as possible, with club owner Marcelo Tinelli not prepared to consider any fee lower than £10million.


text:
We value our editorial independence, basing our comparison results, content and reviews on objective analysis without bias. But we may receive compensation when you click links on our site. Learn more about how we make money from our partners.
How to compare adjustable-rate mortgages
If you want to take advantage of a lower initial rate, then consider an adjustable rate mortgage (ARM)
Last updated:
Commonly referred to as a "variable rate mortgage" or a "floating rate mortgage", an adjustable rate mortgage (ARM) is a loan where the interest rate varies according to an external benchmark (such as the 12 month MTA index which is currently 0.285%). The initial interest rate is typically fixed for a set period after which the interest rate adjusts periodically, such as every month or every six months.
For example, a 2/28 ARM has an initial interest rate that is fixed for two years and then a floating rate for the remaining 28 years of the mortgage.
After the initial rate period ends, the ARM interest rate shifts up and down based on the index, which is another interest rate set by market forces.To determine the ARM rate, the lender takes the index rate and adds on a margin.
Why we like:
A nontraditional lender offering impartial guidance on a range of loans.
- Wide selection of mortgages
- High loan amounts
- No prepayment fee
Why we like: LoanDepot
A nontraditional lender offering impartial guidance on a range of loans.
- Its "no steering" policy holds loan officers to honest, impartial guidance.
- Use online tools to securely verify application information.
- Complete the homebuying process online or find a local loan agent.
- Minimum credit score: 620
Lenders that offer adjustable-rate mortgages
What are the pros and cons for ARMs?
- Lower initial rate. Generally the initial interest rate of an ARM is lower compared to a fixed-rate mortgage. If you only plan to stay in your home for a short amount of time, you can benefit from a lower initial interest rate because you may vacate your home before the initial rate period adjusts.
- Benefit from rate decreases. If interest rates fall, you will benefit from making lower monthly payments which can make your mortgage more affordable.
- Rate cap protection. Many ARMs have an interest rate cap which places a limit on how much the interest rate can increase by, which protects you from steep rate increases.
- Long fixed interest period. If the initial rate period is lengthy, you may be able to treat it as an effective fixed rate. For example, a 7/1 ARM gives you sufficient time to sell the property or refinance without your initial rate ever adjusting. This long fixed interest term means you can save money through lower closing costs throughout the initial period.
- Qualify for larger amount. As the initial payment is often lower compared to a fixed-rate mortgage, an ARM can maximize your home buying power as it can be easier to qualify for a larger loan amount.
- Market risk. After the fixed-rate period ends, the interest rate can rise which could mean that your monthly payments become more expensive after the initial period ends.
- Difficult to budget. As interest rates are unpredictable, you can't predict what your monthly payments will be after the initial period, which can make it difficult to manage your cash flow.
- Difficult to refinance. If interest rates increase, this will not only make your payments more expensive but it will reduce the amount of equity you have in your home which can make it difficult to refinance with another lender.
- Complexity. The structure of ARMs can be difficult to understand for many borrowers when considering the margin, caps, indexes and fees, so unsavvy borrowers may find it difficult to compare ARMs effectively.
Who are ARMs suited to?
Generally ARMs are most suited to borrowers that are:
Comfortable with a degree of risk.
Borrowers must understand that their rate may increase after the initial period and that this will affect their monthly payments. While difficult to predict, an ARM may be useful if you believe interest rates are likely to fall in future.
Holding property for short period.
ARMs are suited to borrowers who only intend to hold their property for a short period of time as this means they can take full advantage of a lower initial rate. ARMs are also useful for borrowers who expect their income to rise in future as this may offset the higher payments following the adjustment period.
Low initial rate benefit.
This type of mortgage is suitable for borrowers who want to benefit primarily from a lower initial rate and lower monthly payments for short-term cash flow benefit.
What are the different types of ARMs?
Hybrid ARMs.
These loans are a mix of fixed-rate period and an adjustable-rate period. The interest rate is fixed for an initial period, such as for 7 years in a 7/1 ARM loan, and after this period, the rate may adjust annually until the loan is paid off.
Interest-only (I-O) ARMs.
An I-O loan allows you to pay only the interest portion of your loan for a set period, normally 3-10 years. This allows you to make smaller monthly payments over an introductory period as you are only paying the interest portion of the payment, and not the principal. After this time, your monthly payment will increase as you start paying back both the principal and interest.
Payment-option ARM.
This is a type of ARM that allows you to select a particular payment option each month, such as a traditional payment of principal and interest, an interest-only payment or a minimum payment. A minimum payment may be less than the amount of interest due that month. Keep in mind that a minimum payment can in turn increase the amount you owe on the loan.
What is the annual cap?
The annual cap restricts the potential increase in the ARM interest rate to a specified amount each year. This is designed to protect borrowers against a substantial increase in payments if the rates increase.
An ARM with an annual cap will only increase by a certain level in terms of percentage points irrespective of how much rates actually rise. For example, a 3% ARM with a 2% cap can only adjust to 5%, even if interest rates increase by 6% during the initial fixed term of the mortgage.
What is the difference between an ARM and a fixed-rate mortgage?
The primary difference between an ARM and a fixed-rate mortgage is that with a fixed-rate mortgage the interest rate remains consistent throughout the life of the loan, while with an ARM the interest rate moves up and down in relation to an index.
In addition, lenders generally charge lower initial interest rates for ARMs compared to fixed-rate mortgages.
How can I interpret ARM rates?
If you have trouble interpreting the different rate periods for hybrid ARMs, keep this in mind: the first number represents how long the fixed interest-rate will be, and the second number represents how often the rate will adjust after the initial period. For example, for a 7/1 ARM, the initial fixed-rate period is 7 years and after this time, the rate will adjust on an annual basis.
You may also see some ARMs which are expressed as 2/28 or 3/27. When represented this way, the first number tells you how many years the initial fixed rate period will be and the second number tells you the number of years the adjustable rate period will be.
Frequently asked questions
Image: ShutterStock
Ask an Expert


text:
Given the dire warnings about climate change, some business and IT people are pondering this question: How should datacentre managers handle the crop of 100- and even 500-year storms, coastal flooding and other ecological disasters that climatologists predict are heading our way?
Some experts suggest that managers of mission-critical IT centres simply need to harden existing facilities, other observers say they need to move the cenres to higher ground and a third group says both strategies are needed.
But one thing is certain, experts say: Few IT organisations -- even those that suffered or narrowly escaped damage from recent major storms -- are thinking long-term. Most IT leaders are, if anything, taking the path of least resistance and expense.
For instance, the datacentre response to Superstorm Sandy, on the East Coast at least, "is nothing more than hardening existing centres," says Peter Sacco, founder and president of PTS Data Center Solutions, a data center design and consultancy firm in Franklin Lakes, N.J. On the other hand, he says, because computers are networked, that deemphasizes "the importance of any single data center."
Internap, a colocation vendor, is strengthening its most at-risk facilities, including the building at 75 Broad Street in downtown Manhattan that flooded after Sandy hit. During the storm, fuel pumps shut down and Internap switched to a 1,200-gallon reserve fuel tank on a higher floor to keep servers running.
"Nobody expected Sandy to become as catastrophic as it was," says Steve Orchard, Internap's senior vice president of development and operations. "With Hurricane Irene the previous year, we're seeing a trend that's a little alarming."
The company has announced it's building a new datacentre in Secaucus, N.J. -- outside the flood plain. "We take climate change very seriously, and it does factor into our new site selection," Orchard says.
Hurricanes Katrina and Rita, followed by Gustav and Ike (2008) and Isaac (2012), slammed into the Gulf Coast with such ferocity that IT executives at Entergy, a $10 billion electrical power company with 14,000 employees, abandoned the idea of a single data center in New Orleans and went back to the drawing board.
Before Katrina hit in 2005, the power company, which provides both nuclear and fossil fuel electricity to 2.8 million customers, had its corporate headquarters in New Orleans and one data center in Gretna, La., just west of downtown New Orleans across the Mississippi River.
"We knew the data center was in the storm's way, and we made a decision after that event to move the data center because we were holding our breath," says Jill Israel, Entergy's CIO. "We didn't have flooding in the immediate area of our data center," she explains. "But there was no power, our lines were down and we had to run on our generator and keep topping it off. "
By winter 2006, the company decided to create two mirror-image data centers in Jackson, Miss., and Little Rock, Ark. In Little Rock, the company retrofitted an old library with sturdy brick walls, moving hardware and critical applications from New Orleans, piece-by-piece, to the backup facility by 2008.
Finally, by 2010, Entergy had completed its brand-new $30 million Jackson, Miss., data center. The company load-balances several systems including email between the two centers, Israel says. "Moving applications from New Orleans involved quite a choreography plan. Subsequent to Katrina we've had [major] storm events, including ice storms in Little Rock. But I no longer have to hold my breath," she says.
The company holds drills for hurricanes and storms every year "to get better and better at responding," Israel says. "One of the things we quickly recognized was how effective a dispersed workforce can be. Our employees can do a lot more things from remote locations and that has served us very, very well."
Not everyone seems to have absorbed the "take heed" message. IT shops in both Europe and America's Northeast seem to cling to the idea that superstorms are non-repeatable freaks of nature. In some cases, even among those affected by major storms, vigilance plays a chess game with artful forgetfulness as managers choose other IT priorities.
Story continues on next page
In Florida, replication is key
Flooding, high winds and hurricanes are yearly occurrences in Florida, and data center backup has become key to local government continuing to function during disasters. Longboat Key is a barrier island with a winter population of 7,000 people that swells to 22,000 in the summer. Located on the west coast between Tampa and Fort Myers, the town decided to replicate critical data services in a secure mainland facility in 2004.
"Florida was just inundated with hurricanes that year, so everyone fortified their disaster recovery plans and efforts," says Kathi Pletzke, Longboat Key's IT director. "We had to assure that if a disaster were to occur, such as a hurricane, we'd need to set up shop fairly quickly and start serving the needs of citizens."
Longboat Key falls geographically into both Sarasota and Manatee counties. Tom Warner, Sarasota County's disaster recovery administrator, began conversations in 2004 with IT leaders in several western counties throughout the state. The result: Warner co-founded the Florida Technology Disaster Recovery Consortium, with Kevin Kryzda, CIO of Martin County, to help one another with replication throughout the region.
"The thought process was: How can we help each other because we are in Hurricane Alley, and what can we put in place long before hurricane season starts to ensure critical applications can be safe?" Warner explains. They looked at data centers to see which were hardened, which category of hurricane they could withstand and if they had additional floor space to house the counties' servers in their location.
With Sarasota County in the lead, the group eventually brokered a deal with Verizon to secure rack and server space for several counties in one of Verizon's switching centers, a secure brick building with no windows located near a major highway and around 10 miles inland from Longboat Key.
The town has replicated its servers at Suncoast Technology Center -- the former Verizon switching center -- and "though we don't have everything on it, as budgets allow, we've replicated the critical applications, including a mirror image of our production server, payroll, general ledger and public safety data," Pletzke says.
Because there are a lot of elderly people in Florida, "we wanted to replicate our website in [adjoining] Martin County so that in the event of disaster, people could check up on their parents in Sarasota County and see that they are well," Warner says. The consortium has also boosted cooperation among many counties to ensure that IT managers can get emergency access to guest servers and generators to keep computers running smoothly.
-- Arielle Emmett
Ignorance isn't bliss
"What was perceived as a safe area before may not be now," says Rakesh Kumar, a Gartner vice president who specializes in data center and infrastructure issues. In Europe, especially, he cites freezing temperatures, coastal flooding and other unpredictable weather events. In Asia, tsunamis are a concern. "Until we have a major data outage, though, most clients are not calculating for risk or change; they're turning a blind eye to it," Kumar says.
Many of his European and U.S. clients praise the idea of doing thorough risk assessments and thinking proactively, long-term, he says. At least in theory.
But even now, six months after Sandy, most East Coast-based companies aren't being proactive about repositioning their data centers, experts say. "They're expanding in the same locations; they're not even thinking about moving," says Neil Sheehan, a data center architect and principal of Chicago-based Sheehan Partners Ltd.
In fact, he says, "we are looking for expansion for our clients in New Jersey right near the coast, [near] sites that flooded." Sheehan says with proper surveys of 500-year-flood levels, data center architects can determine the ideal height of first floors, so that flooding, if it occurs, happens in parking lots and not in data centers.
Some are paying more than lip service. In downtown Manhattan, at 140 West Street, a Verizon switching center felt the full force of Sandy's stormwater. Five sub-basement levels, including a Verizon cable vault, went underwater. Technicians struggled to mount emergency generators and pump water out through elevator shafts.
Since then, Verizon has had to extract 150 tons of damaged copper cable from lower Manhattan streets, its central office, headquarters and customer sites, replacing virtually all of it with weather-proof fiber cable protected in conduit. "If you take a fiber optic cable and lay it in your bathtub it probably will still work; fiber is submersible," says Chris Kimm, vice president of global customer assurance for Verizon Enterprise Solutions.
By replacing electrical infrastructure and bringing it up to higher floors, the carrier was able to get its own central office back online in about a week.
Some customers are following suit -- moving critical infrastructure to floors that once housed rentable office space. And some are deploying new services, Kimm says, including mobile wireless, wireless IP and cloud computing solutions to allow their employees to work remotely. Others are rerouting their telecom and data networks. "We even have some buildings that landlords have to redo, converting them from business locations to residences and deploying wireless services."
Even given all the damage, however, Verizon isn't considering moving its switching centers to a less flood-prone location. Instead, Kimm says, "We're armoring the buildings; we've done an evaluation of what all the risks are," he says. "We haven't gotten final solutions, but the start of the next hurricane season [is a few months off]; you've got until mid-summer before you have a significant risk of a future event."
Lessons learned?
The impact of climate change and storms like Katrina and Sandy remains difficult to calculate. Not even climatologists can predict the frequency of extreme weather events as ocean levels and temperatures rise. But in the U.S., locations such as Manhattan, Long Island, New Jersey, Miami, Virginia Beach, Boston, Washington, D.C., and even Seattle and San Diego are expected to see increased coastal flooding.
"I think it's absolutely compelling to look at the impact of recent storms, and also to look at statistics that show there have been more natural [severe] weather events, whether that's related to global climate shifts or other factors," says Jim Grogan, a business continuity and resilience specialist at 451 Research.
"Every single event, though, leaves lessons to be learned," he says. "Lessons come from the stories of the creative and innovative things data center operators did to keep their centers" going even in the worst conditions.
Though Sandy may have been a wake-up call for major data centers in the New York area to take some steps to harden facilities, it remains unclear how many will act on a longer-term solution -- moving out of the city entirely, for example, or developing redundant and geographically separate facilities, or opting for third-party disaster recovery and cloud solutions.
New thinking is happening among some of the larger IT organizations. "We're definitely seeing companies looking at alternative locations for data center operations," says 451's Grogan. "We have spoken to multitenant data center operators in Atlanta, Virginia and other locations, and we're seeing an increase of interest from customers in the Northeast."
Many businesses that can afford a disaster recovery solution are looking now toward secured multitenant providers outside major cities. Cervalis, for example, a company with more than 200 large corporate clients including global banks and software and testing companies, has hundreds of thousands of square feet of multitenant space in upstate New York, Passaic County, N.J., and Fairfield County, Conn., all safely tucked away from floodplains.
The company acts as a primary or secondary data center, providing secured cabinets and redundant power, fully loaded (20 to 40 servers per cabinet), for about $1,500 per month per cabinet. That price is still cheaper for companies than building a redundant data facility on their own, insists Zack Margolis, a Cervalis vice president of sales, marketing and business development.
But not all businesses can afford this level of protection. Small businesses might realize the importance of data backups, but most will have to avail themselves of less expensive forms of resilience, such as backup tapes mounted at small disaster recovery facilities or virtualized clouds.
Moreover, many businesses still cling to major cities as the center of the universe. "Downtown Manhattan will still be in high demand because it is an interconnection hub to the United States," says 451's Michael Levy. Further, financial companies still want to "touch their data" and have it near the center of trading action without fear of latency, even though fiber-connected facilities outside the city can do the same job.
Overall, Gartner's Kumar insists, too few IT leaders are taking the signs of weather and climate change seriously enough. "We have had cases of coastal [flooding] where climate change has become an issue," he says. "In London and Germany, the winters seem to be getting slightly worse; we've had cases of component failure -- small bits of electrical equipment freezing up."
Despite all this, most IT managers still aren't willing to make proactive risk assessments to avert disasters. "When I mention risk assessments, they say, 'Rakesh, great point. We'll get engineering to complete a report.' But two months later, it's still at the bottom of the 'to-do' pile."
How to keep going in the next freak storm
You can do many things to plan ahead of a storm. Here are the top picks from those who learned the hard way during Superstorm Sandy.
Secure fuel tanks and pumps. In Manhattan, most downtown data center outages occurred due to ruptured fuel pumps, tanks and shorted electrical wiring. At Datagram -- a private co-location company providing managed services to Gawker, Buzzfeed and The Huffington Post, among others -- torrents of water in the basement blew off stainless steel doors, rupturing a diesel fuel tank. "Our generators were operational, but we had pumps submerged under saltwater, which caused a failure of our fuel systems," says Alex Reppen, Datagram's CEO. The lesson: Install multiple fuel tanks and encase them in concrete so they can't be damaged in a storm surge.
Check emergency wiring, hoses, connectors and back-up circuits. At Peer1 Hosting, with 21 data centers around the country, human bucket brigades brought needed diesel fuel up to a rooftop generator as multiple shifts of people worked overnight to keep a Manhattan data center online. However, "we had a heat issue and lost a portion of our data center for a day because we couldn't [reduce] the heat during the power outage," explains Ted Smith, senior vice president of Peer1 Hosting. Lessons: Review backup power plans, check UPS batteries and make sure there are enough emergency wiring, hoses, connectors and fuel to run backup equipment and generators for prolonged periods.
Get creative in the basement. When basement floors flood, fuel tanks can float, but they also do damage to pipes and fuel pumps. That was a big problem for Internap during Sandy, and New York City regulations require that fuel tanks remain in city basements. "The challenge," says Internap's Steve Orchard, "is that fuel tanks may be underground, somewhat buried, or sitting above ground in basements on stands depending on when the building was built." Lessons: Make tanks submersible by strapping them into place to structural steel so they won't float, Orchard says. "We're also looking into submersible pumps sitting inside the fuel tank along with placing a redundant pump on the mezzanine level." For added safety, Internap could also install a basement pump at a height above six feet.
Consider triple power backup. New York's CitiServ data center in Brooklyn -- a two-building project that is part of Mayor Michael Bloomberg's efforts to consolidate 55 city data centers into one -- stayed up throughout Sandy while managing the emergency 911 and 311 services. "We placed the facility in Brooklyn outside the hurricane flood zone, and we had a triple power backup strategy," which includes utility power, emergency generators and battery, says Rahul N. Merchant, New York's chief information and innovation officer. At the height of the storm, lights flickered briefly, but generator power kicked in seamlessly. A tank holding 200,000 gallons of diesel located on-site at the Brooklyn facility -- and topped off regularly -- powered everything for 45.5 hours before regular utility power was restored.
Re-patch and restore. In addition, the city's own fiber optic network, CitiNet, was used to restore 311 service when Verizon's cable vaults flooded and failed at downtown Manhattan nodes. "We were able to re-patch the 311 network by using our own fiber backbone and connecting it [directly] into the Verizon Central Office in Brooklyn," Merchant says.
Use sandbags. The bullet just missed when Sandy hammered at QTS's data center in Jersey City, N.J. "We were impacted by rising waters and high winds, and the water pushed up right to the edges of our building," says Brian Johnston, QTS's chief technology officer. However, the building was storm-proofed; the actual data center had been located on the 16th through 20th floors, with all power generation and logic on the same level except for electrical vaults and lines coming from the streets. "In the event we lose the vaults, everything we need transfers to UPS; and our fuel oil in 1,000-gallon fuel tanks along with electric pumps in the garage parking lot are also water-proofed," Johnston says. Though QTS temporarily lost utility power, backup generators took over, and no basement flooding occurred. Lesson: "Our skilled staff redirected waters with sand bags. We had wraps around any openings in the building and our parking center, so water couldn't come into parking garage or flood our vaults." Future plans: more flood blockers -- borrowed from hurricane protocols in Miami -- and training for IT staff in emergency procedures.