text:
Data on the Outside vs. Data on the Inside
Data kept outside SQL has different characteristics from data kept inside.
The History, Status, and Future of FPGAs
Hitting a nerve with field-programmable gate arrays
Debugging Incidents in Google's Distributed Systems
How experts debug production issues in complex distributed systems
Is Persistent Memory Persistent?
A simple and inexpensive test of failure-atomic update mechanisms
Dark Patterns: Past, Present, and Future
The evolution of tricky user interfaces
Demystifying Stablecoins
Cryptography meets monetary policy
Beyond the Fix-it Treadmill
The Use of Post-Incident Artifacts in High-Performing Organizations
Managing the Hidden Costs of Coordination
Controlling coordination costs when multiple, distributed perspectives are essential
Cognitive Work of Hypothesis Exploration During Anomaly Response
A look at how we respond to the unexpected
Above the Line, Below the Line
The resilience of Internet-facing systems relies on what is below the line of representation.
Revealing the Critical Role of Human Performance in Software
It's time to revise our appreciation of the human side of Internet-facing software systems.
Blockchain Technology: What Is It Good for?
Industry's dreams and fears for this new technology
The Reliability of Enterprise Applications
Understanding enterprise reliability
Optimizations in C++ Compilers
A practical journey
Hack for Hire
Investigating the emerging black market of retail email account hacking services
The Effects of Mixing Machine Learning and Human Judgment
Collaboration between humans and machines does not necessarily lead to better outcomes.
Persistent Memory Programming on Conventional Hardware
The persistent memory style of programming can dramatically simplify application software.
Velocity in Software Engineering
From tectonic plate to F-16
Open-source Firmware
Step into the world behind the kernel.
Surviving Software Dependencies
Software reuse is finally here but comes with risks.
Industry-scale Knowledge Graphs: Lessons and Challenges
Five diverse technology companies show how it's done
Garbage Collection as a Joint Venture
A collaborative approach to reclaiming memory in heterogeneous software systems
Online Event Processing
Achieving consistency where distributed transactions have failed
Net Neutrality: Unexpected Solution to Blockchain Scaling
Cloud-delivery networks could dramatically improve blockchains' scalability, but clouds must be provably neutral first.
Identity by Any Other Name
The complex cacophony of intertwined systems
Achieving Digital Permanence
The many challenges to maintaining stored information and ways to overcome them
Metrics That Matter
Critical but oft-neglected service metrics that every SRE and product owner should care about
A Hitchhiker's Guide to the Blockchain Universe
Blockchain remains a mystery, despite its growing acceptance.
Tear Down the Method Prisons! Set Free the Practices!
Essence: a new way of thinking that promises to liberate the practices and enable true learning organizations
Understanding Database Reconstruction Attacks on Public Data
These attacks on statistical databases are no longer a theoretical danger.
Using Remote Cache Service for Bazel
Save time by sharing and reusing build and test output
Why SRE Documents Matter
How documentation enables SRE teams to manage new and existing services
How to Live in a Post-Meltdown and -Spectre World
Learn from the past to prepare for the next battle.
Tracking and Controlling Microservice Dependencies
Dependency management is a crucial part of system and software design.
Corp to Cloud: Google's Virtual Desktops
How Google moved its virtual desktops to the cloud
The Mythos of Model Interpretability
In machine learning, the concept of interpretability is both important and slippery.
Mind Your State for Your State of Mind
The interactions between storage and applications can be complex and subtle.
Workload Frequency Scaling Law - Derivation and Verification
Workload scalability has a cascade relation via the scale factor.
Algorithms Behind Modern Storage Systems
Different uses for read-optimized B-trees and write-optimized LSM-trees
C Is Not a Low-level Language
Your computer is not a fast PDP-11.
Thou Shalt Not Depend on Me
A look at JavaScript libraries in the wild
Designing Cluster Schedulers for Internet-Scale Services
Embracing failures for improving availability
Canary Analysis Service
Automated canarying quickens development, improves production safety, and helps prevent outages.
Continuous Delivery Sounds Great, but Will It Work Here?
It's not magic, it just requires continuous, daily improvement at all levels.
Containers Will Not Fix Your Broken Culture (and Other Hard Truths)
Complex socio-technical systems are hard; film at 11.
DevOps Metrics
Your biggest mistake might be collecting the wrong data.
Monitoring in a DevOps World
Perfect should never be the enemy of better.
Bitcoin's Underlying Incentives
The unseen economic forces that govern the Bitcoin protocol
Titus: Introducing Containers to the Netflix Cloud
Approaching container adoption in an already cloud-native infrastructure
Abstracting the Geniuses Away from Failure Testing
Ordinary users need tools that automate the selection of custom-tailored faults to inject.
Network Applications Are Interactive
The network era requires new models, with interactions instead of algorithms.
Cache Me If You Can
Building a decentralized web-delivery model
Bitcoin's Academic Pedigree
The concept of cryptocurrencies is built from forgotten ideas in research literature.
Metaphors We Compute By
Code is a story that explains how to solve a particular problem.
Is There a Single Method for the Internet of Things?
Essence can keep software development for the IoT from becoming unwieldy.
Data Sketching
The approximate approach is often faster and more efficient.
The Calculus of Service Availability
You're only as available as the sum of your dependencies.
The IDAR Graph
An improvement over UML
Too Big NOT to Fail
Embrace failure so it doesn't embrace you.
The Debugging Mindset
Understanding the psychology of learning strategies leads to effective problem-solving skills.
MongoDB's JavaScript Fuzzer
The fuzzer is for those edge cases that your testing didn't catch.
Making Money Using Math
Modern applications are increasingly using probabilistic machine-learned models.
Pervasive, Dynamic Authentication of Physical Items
The use of silicon PUF circuits
Uninitialized Reads
Understanding the proposed revisions to the C language
Heterogeneous Computing: Here to Stay
Hardware and Software Perspectives
Time, but Faster
A computing adventure about time through the looking glass
Life Beyond Distributed Transactions
An apostate's opinion
BBR: Congestion-Based Congestion Control
Measuring bottleneck bandwidth and round-trip propagation time
Faucet: Deploying SDN in the Enterprise
Using OpenFlow and DevOps for rapid development
Industrial Scale Agile - from Craft to Engineering
Essence is instrumental in moving software development toward a true engineering discipline.
Functional at Scale
Applying functional programming principles to distributed computing projects
Scaling Synchronization in Multicore Programs
Advanced synchronization methods can boost the performance of multicore software.
Idle-Time Garbage-Collection Scheduling
Taking advantage of idleness to reduce dropped frames and memory consumption
Dynamics of Change: Why Reactivity Matters
Tame the dynamics of change by centralizing each concern in its own module.
Cluster-level Logging of Containers with Containers
Logging Challenges of Container-Based Cloud Deployments
The Hidden Dividends of Microservices
Microservices aren't for every company, and the journey isn't easy.
Debugging Distributed Systems
Challenges and options for validation and debugging
Should You Upload or Ship Big Data to the Cloud?
The accepted wisdom does not always hold true.
The Flame Graph
This visualization of software execution is a new necessity for performance profiling and debugging.
Why Logical Clocks are Easy
Sometimes all you need is the right language.
Use-Case 2.0
The Hub of Software Development
Statistics for Engineers
Applying statistical techniques to operations data
Borg, Omega, and Kubernetes
Lessons learned from three container-management systems over a decade
The Verification of a Distributed System
A practitioner's guide to increasing confidence in system correctness
Accountability in Algorithmic Decision-making
A view from computational journalism
Immutability Changes Everything
We need it, we can afford it, and the time is now.
Time is an Illusion.
Lunchtime doubly so. - Ford Prefect to Arthur Dent in "The Hitchhiker's Guide to the Galaxy", by Douglas Adams
Non-volatile Storage
Implications of the Datacenter's Shifting Center
Schema.org: Evolution of Structured Data on the Web
Big data makes common schemas even more necessary.
A Purpose-built Global Network: Google's Move to SDN
A discussion with Amin Vahdat, David Clark, and Jennifer Rexford
It Probably Works
Probabilistic algorithms are all around us--not only are they acceptable, but some programmers actually seek out chances to use them.
Challenges of Memory Management on Modern NUMA System
Optimizing NUMA systems applications with Carrefour
Componentizing the Web
We may be on the cusp of a new revolution in web development.
Fail at Scale
Reliability in the face of rapid change
How to De-identify Your Data
Balancing statistical accuracy and subject privacy in large social-science data sets
Crash Consistency
Rethinking the Fundamental Abstractions of the File System
Testing a Distributed System
Testing a distributed system can be trying even under the best of circumstances.
Natural Language Translation at the Intersection of AI and HCI
Old questions being answered with both AI and HCI
Beyond Page Objects: Testing Web Applications with State Objects
Use states to drive your tests
Dismantling the Barriers to Entry
We have to choose to build a web that is accessible to everyone.
Hadoop Superlinear Scalability
The perpetual motion of parallel performance
Evolution and Practice: Low-latency Distributed Applications in Finance
The finance industry has unique demands for low-latency distributed systems.
The Science of Managing Data Science
Lessons learned managing a data science research team
Using Free and Open Source Tools to Manage Software Quality
An agile process implementation
From the EDVAC to WEBVACs
Cloud computing for computer scientists
Spicing Up Dart with Side Effects
A set of extensions to the Dart programming language, designed to support asynchrony and generator functions
Reliable Cron across the Planet
...or How I stopped worrying and learned to love time
There is No Now
Problems with simultaneity in distributed systems
Parallel Processing with Promises
A simple method of writing a collaborative system
META II: Digital Vellum in the Digital Scriptorium
Revisiting Schorre's 1962 compiler-compiler
Model-based Testing: Where Does It Stand?
MBT has positive effects on efficiency and effectiveness, even if it only partially fulfills high expectations.
Go Static or Go Home
In the end, dynamic systems are simply less secure.
Securing the Network Time Protocol
Crackers discover how to use NTP as a weapon for abuse.
Scalability Techniques for Practical Synchronization Primitives
Designing locking primitives with performance in mind
Internal Access Controls
Trust, but Verify
Disambiguating Databases
Use the database built for your access model.
A New Software Engineering
What happened to the promise of rigorous, disciplined, professional practices for software development?
There's No Such Thing as a General-purpose Processor
And the belief in such a device is harmful
The Responsive Enterprise: Embracing the Hacker Way
Soon every company will be a software company.
Evolution of the Product Manager
Better education needed to develop the discipline
Productivity in Parallel Programming: A Decade of Progress
Looking at the design and benefits of X10
JavaScript and the Netflix User Interface
Conditional dependency resolution
Security Collapse in the HTTPS Market
Assessing legal and technical solutions to secure HTTPS
Why Is It Taking So Long to Secure Internet Routing?
Routing security incidents can still slip past deployed security defenses.
Certificate Transparency
Public, verifiable, append-only logs
Securing the Tangled Web
Preventing script injection vulnerabilities through software design
Privacy, Anonymity, and Big Data in the Social Sciences
Quality social science research and the privacy of human subjects requires trust.
The Network is Reliable
An informal survey of real-world communications failures
Undergraduate Software Engineering: Addressing the Needs of Professional Software Development
Addressing the Needs of Professional Software Development
Bringing Arbitrary Compute to Authoritative Data
Many disparate use cases can be satisfied with a single storage system.
Who Must You Trust?
You must have some trust if you want to get anything done.
Automated QA Testing at EA: Driven by Events
A discussion with Michael Donat, Jafar Husain, and Terry Coatta
Design Exploration through Code-generating DSLs
High-level DSLs for low-level programming
Finding More Than One Worm in the Apple
If you see something, say something.
Domain-specific Languages and Code Synthesis Using Haskell
Looking at embedded DSLs
The NSA and Snowden: Securing the All-Seeing Eye
How good security at the NSA could have stopped him
The Curse of the Excluded Middle
"Mostly functional" programming does not work.
Don't Settle for Eventual Consistency
Stronger properties for low-latency geo-replicated storage
A Primer on Provenance
Better understanding of data requires tracking its history and context.
Multipath TCP
Decoupled from IP, TCP is at last able to support multihomed hosts.
Major-league SEMAT: Why Should an Executive Care?
Becoming better, faster, cheaper, and happier
Eventually Consistent: Not What You Were Expecting?
Methods of quantifying consistency (or lack thereof) in eventually consistent storage systems
Scaling Existing Lock-based Applications with Lock Elision
Lock elision enables existing lock-based programs to achieve the performance benefits of nonblocking synchronization and fine-grain locking with minor software engineering effort.
Rate-limiting State
The edge of the Internet is an unruly place
The API Performance Contract
How can the expected interactions between caller and implementation be guaranteed?
Provenance in Sensor Data Management
A cohesive, independent solution for bringing provenance to scientific research
Unikernels: Rise of the Virtual Library Operating System
What if all the software layers in a virtual appliance were compiled within the same safe, high-level language framework?
Toward Software-defined SLAs
Enterprise computing in the public cloud
The Road to SDN
An intellectual history of programmable networks
The Software Inferno
Dante's tale, as experienced by a software architect
Making the Web Faster with HTTP 2.0
HTTP continues to evolve
Intermediate Representation
The increasing significance of intermediate representations in compilers
The Challenge of Cross-language Interoperability
Interfacing between languages is increasingly important.
Agile and SEMAT - Perfect Partners
Combining agile and SEMAT yields more advantages than either one alone
Adopting DevOps Practices in Quality Assurance
Merging the art and science of software development
Passively Measuring TCP Round-trip Times
A close look at RTT measurements with TCP
Leaking Space
Eliminating memory hogs
Barbarians at the Gateways
High-frequency Trading and Exchange Technology
Online Algorithms in High-frequency Trading
The challenges faced by competing HFT algorithms
The Balancing Act of Choosing Nonblocking Features
Design requirements of nonblocking systems
NUMA (Non-Uniform Memory Access): An Overview
NUMA becomes more common because memory controllers get close to execution units on microprocessors.
20 Obstacles to Scalability
Watch out for these pitfalls that can prevent Web application scaling.
Rules for Mobile Performance Optimization
An overview of techniques to speed page loading
Best Practices on the Move: Building Web Apps for Mobile Devices
Which practices should be modified or avoided altogether by developers for the mobile Web?
The Antifragile Organization
Embracing Failure to Improve Resilience and Maximize Availability
Nonblocking Algorithms and Scalable Multicore Programming
Exploring some alternatives to lock-based synchronization
Proving the Correctness of Nonblocking Data Structures
So you've decided to use a nonblocking data structure, and now you need to be certain of its correctness. How can this be achieved? When a multithreaded program is too slow because of a frequently acquired mutex, the programmer's typical reaction is to question whether this mutual exclusion is indeed required. This doubt becomes even more pronounced if the mutex protects accesses to only a single variable performed using a single instruction at every site. Removing synchronization improves performance, but can it be done without impairing program correctness?
Structured Deferral: Synchronization via Procrastination
We simply do not have a synchronization mechanism that can enforce mutual exclusion.
Realtime GPU Audio
Finite difference-based sound synthesis using graphics processors
There's Just No Getting around It: You're Building a Distributed System
Building a distributed system requires a methodical approach to requirements.
A File System All Its Own
Flash memory has come a long way. Now it's time for software to catch up.
Eventual Consistency Today: Limitations, Extensions, and Beyond
How can applications be built on eventually consistent infrastructure given no guarantee of safety?
Discrimination in Online Ad Delivery
Google ads, black names and white names, racial discrimination, and click advertising
How Fast is Your Web Site?
Web site performance data has never been more readily available.
FPGA Programming for the Masses
The programmability of FPGAs must improve if they are to be part of mainstream computing.
The Evolution of Web Development for Mobile Devices
Building Web sites that perform well on mobile devices remains a challenge.
The Story of the Teapot in DHTML
It's easy to do amazing things, such as rendering the classic teapot in HTML and CSS.
Making the Mobile Web Faster
Mobile performance issues? Fix the back end, not just the client.
Hazy: Making it Easier to Build and Maintain Big-data Analytics
Racing to unleash the full potential of big data with the latest statistical and machine-learning techniques.
A Decade of OS Access-control Extensibility
Open source security foundations for mobile and embedded devices
Rethinking Passwords
Our authentication system is lacking. Is improvement possible?
Thinking Methodically about Performance
The USE method addresses shortcomings in other commonly used methodologies.
Splinternet Behind the Great Firewall of China
Once China opened its door to the world, it could not close it again.
Condos and Clouds
Constraints in an environment empower the services.
The Web Won't Be Safe or Secure until We Break It
Unless you've taken very particular precautions, assume every Web site you visit knows exactly who you are.
The Essence of Software Engineering: The SEMAT Kernel
A thinking framework in the form of an actionable kernel
Anatomy of a Solid-state Drive
While the ubiquitous SSD shares many features with the hard-disk drive, under the surface they are completely different.
Sender-side Buffers and the Case for Multimedia Adaptation
A proposal to improve the performance and availability of streaming video and other time-sensitive media
Weathering the Unexpected
Failures happen, and resilience drills help organizations prepare for them.
Disks from the Perspective of a File System
Disks lie. And the controllers that run them are partners in crime.
Toward Higher Precision
An introduction to PTP and its significance to NTP practitioners
Fault Injection in Production
Making the case for resilience testing
All Your Database Are Belong to Us
In the big open world of the cloud, highly available distributed objects will rule.
Software Needs Seatbelts and Airbags
Finding and fixing bugs in deployed software is difficult and time-consuming. Here are some alternatives.
A New Objective-C Runtime: from Research to Production
Backward compatibility always trumps new features.
Multitier Programming in Hop
A first step toward programming 21st-century applications
OpenFlow: A Radical New Idea in Networking
An open standard that enables software-defined networking
Extending the Semantics of Scheduling Priorities
Increasing parallelism demands new paradigms.
Getting What You Measure
Four common pitfalls in using software metrics for project management
Modeling People and Places with Internet Photo Collections
Understanding the world from the sea of online photos
Controlling Queue Delay
A modern AQM is just one piece of the solution to bufferbloat.
A Guided Tour through Data-center Networking
A good user experience depends on predictable performance within the data-center network.
Realtime Computer Vision with OpenCV
Mobile computer-vision technology will soon become as ubiquitous as touch interfaces.
Idempotence Is Not a Medical Condition
An essential property for reliable systems
CPU DB: Recording Microprocessor History
With this open database, you can mine microprocessor trends over the past 40 years.
Your Mouse is a Database
Web and mobile applications are increasingly composed of asynchronous and realtime streaming services and push notifications.
Managing Technical Debt
Shortcuts that save money and time today can cost you down the road.
Interactive Dynamics for Visual Analysis
A taxonomy of tools that support the fluent and flexible use of visualizations
Why LINQ Matters:
Cloud Composability Guaranteed
The benefits of composability are becoming clear in software engineering.
Revisiting Network I/O APIs: The netmap Framework
It is possible to achieve huge performance improvements in the way packet processing is done on modern operating systems.
SAGE: Whitebox Fuzzing for Security Testing
SAGE has had a remarkable impact at Microsoft.
You Don't Know Jack about Shared Variables or Memory Models
Data races are evil.
Advances and Challenges in Log Analysis
Logs contain a wealth of information for help in managing systems.
Bufferbloat: Dark Buffers in the Internet
Networks without effective AQM may again be vulnerable to congestion collapse.
I/O Virtualization
Decoupling a logical device from its physical implementation offers many compelling advantages.
Creating Languages in Racket
Sometimes you just have to make a better mousetrap.
Coding Guidelines:
Finding the Art in the Science
What separates good code from great code?
How Will Astronomy Archives Survive the Data Tsunami?
Astronomers are collecting more data than ever. What practices can keep them ahead of the flood?
Postmortem Debugging in Dynamic Environments
Modern dynamic languages lack tools for understanding software failures.
OCaml for the Masses
Why the next language you learn should be functional
Java Security Architecture Revisited
Hard technical problems and tough business challenges
The World According to LINQ
Big data is about more than size, and LINQ is more than up to the task.
Verification of Safety-critical Software
Avionics software safety certification is achieved through objective-based standards.
Abstraction in Hardware System Design
Applying lessons from software languages to hardware languages using Bluespec SystemVerilog
Arrogance in Business Planning
Technology business plans that assume no competition (ever)
The Pain of Implementing LINQ Providers
It's no easy task for NoSQL
Computing without Processors
Heterogeneous systems allow us to target our programming to the appropriate environment.
The Robustness Principle Reconsidered
Seeking a middle ground
DSL for the Uninitiated
Domain-specific languages bridge the semantic gap in programming.
If You Have Too Much Data, then "Good Enough" Is Good Enough
In today's humongous database systems, clarity may be relaxed, but business needs can still be met.
Passing a Language through the Eye of a Needle
How the embeddability of Lua impacted its design
Scalable SQL
How do large-scale sites and applications remain SQL-based?
Mobile Application Development: Web vs. Native
Web apps are cheaper to develop and deploy than native apps, but can they match the native user experience?
Weapons of Mass Assignment
A Ruby on Rails app highlights some serious, yet easily avoided, security vulnerabilities.
A co-Relational Model of Data for Large Shared Data Banks
Contrary to popular belief, SQL and noSQL are really just two sides of the same coin.
Successful Strategies for IPv6 Rollouts. Really.
Knowing where to begin is half the battle.
Returning Control to the Programmer:
SIMD Intrinsics for Virtual Machines
Exposing SIMD units within interpreted languages could simplify programs and unleash floods of untapped processor power.
Testable System Administration
Models of indeterminism are changing IT management.
National Internet Defense - Small States on the Skirmish Line
Attacks in Estonia and Georgia highlight key vulnerabilities in national Internet infrastructure.
Finding Usability Bugs with Automated Tests
Automated usability tests can be valuable companions to in-person tests.
System Administration Soft Skills
How can system administrators reduce stress and conflict in the workplace?
A Plea to Software Vendors from Sysadmins - 10 Do's and Don'ts
What can software vendors do to make the lives of sysadmins a little easier?
Collaboration in System Administration
For sysadmins, solving problems usually involves collaborating with others. How can we make it more effective?
Virtualization: Blessing or Curse?
Managing virtualization at a large scale is fraught with hidden challenges.
The Case Against Data Lock-in
Want to keep your users? Just make it easy for them to leave.
Keeping Bits Safe:
How Hard Can It Be?
As storage systems grow larger and larger, protecting their data for long-term storage is becoming more and more challenging.
Tackling Architectural Complexity with Modeling
Component models can help diagnose architectural problems in both new and existing systems.
Thinking Clearly about Performance
Improving the performance of complex software is difficult, but understanding some fundamental principles can make it easier.
Computers in Patient Care: The Promise and the Challenge
Information technology has the potential to radically transform health care. Why has progress been so slow?
Injecting Errors for Fun and Profit
Error-detection and correction features are only as good as our ability to test them.
Lessons from the Letter
Security flaws in a large organization
Software Development with Code Maps
Could those ubiquitous hand-drawn code diagrams become a thing of the past?
The Ideal HPC Programming Language
Maybe it's Fortran. Or maybe it just doesn't matter.
Visualizing System Latency
Heat maps are a unique and powerful way to visualize latency data. Explaining the results, however, is an ongoing challenge.
A Tour through the Visualization Zoo
A survey of powerful visualization techniques, from the obvious to the obscure
Securing Elasticity in the Cloud
Elastic computing has great potential, but many security challenges remain.
Principles of Robust Timing over the Internet
The key to synchronizing clocks over networks is taming delay variability.
Why Cloud Computing Will Never Be Free
The competition among cloud providers may drive prices downward, but at what cost?
Simplicity Betrayed
Emulating a video system shows how even a simple interface can be more complex—and capable—than it appears.
Enhanced Debugging with Traces
An essential technique used in emulator development is a useful addition to any programmer's toolbox.
Cooling the Data Center
What can be done to make cooling systems in data centers more energy efficient?
Toward Energy-Efficient Computing
What will it take to make server-side computing more energy efficient?
Managing Contention for Shared Resources on Multicore Processors
Contention for caches, memory controllers, and interconnects can be alleviated by contention-aware scheduling algorithms.
Power-Efficient Software
Power-manageable hardware can help save energy, but what can software developers do to address the problem?
Triple-Parity RAID and Beyond
As hard-drive capacities continue to outpace their throughput, the time has come for a new level of RAID.
Data in Flight
How streaming SQL technology can help solve the Web 2.0 data crunch.
Maximizing Power Efficiency with Asymmetric Multicore Systems
Asymmetric multicore systems promise to use a lot less energy than conventional symmetric processors. How can we develop software that makes the most out of this potential?
Other People's Data
Companies have access to more types of external data than ever before. How can they integrate it most effectively?
What DNS Is Not
DNS is many things to many people - perhaps too many things to too many people.
You Don't Know Jack About Software Maintenance
Long considered an afterthought, software maintenance is easiest and most effective when built into a system from the ground up.
Metamorphosis: the Coming Transformation of Translational Systems Biology
In the future computers will mine patient data to deliver faster, cheaper healthcare, but how will we design them to give informative causal explanations? Ideas from philosophy, model checking, and statistical testing can pave the way for the needed translational systems biology.
Probing Biomolecular Machines with Graphics Processors
The evolution of GPU processors and programming tools is making advanced simulation and analysis techniques accessible to a growing community of biomedical scientists.
Unifying Biological Image Formats with HDF5
The biosciences need an image format capable of high performance and long-term maintenance. Is HDF5 the answer?
A Threat Analysis of RFID Passports
Do RFID passports make us vulnerable to identity theft?
Communications Surveillance:
Privacy and Security at Risk
As the sophistication of wiretapping technology grows, so too do the risks it poses to our privacy and security.
Four Billion Little Brothers?:
Privacy, mobile phones, and ubiquitous data collection
Participatory sensing technologies could improve our lives and our communities, but at what cost to our privacy?
Making Sense of Revision-control Systems
Whether distributed or centralized, all revision-control systems come with complicated sets of tradeoffs. How do you find the best match between tool and team?
Monitoring and Control of Large Systems with MonALISA
MonALISA developers describe how it works, the key design principles behind it, and the biggest technical challenges in building it.
The Pathologies of Big Data
Scale up your datasets enough and all your apps will come undone. What are the typical problems and where do the bottlenecks generally surface?
Browser Security:
Lessons from Google Chrome
Google Chrome developers focused on three key problems to shield the browser from attacks.
Whither Sockets?
High bandwidth, low latency, and multihoming challenge the sockets API.
Network Front-end Processors, Yet Again
The history of NFE processors sheds light on the tradeoffs involved in designing network stack software.
Fighting Physics: A Tough Battle
Thinking of doing IPC over the long haul? Think again. The laws of physics say you're hosed.
Cybercrime 2.0: When the Cloud Turns Dark
Web-based malware attacks are more insidious than ever. What can be done to stem the tide?
How Do I Model State? Let Me Count the Ways
A study of the technology and sociology of Web services specifications
Security in the Browser
Web browsers leave users vulnerable to an ever-growing number of attacks. Can we make them secure while preserving their usability?
The Obama Campaign:
A Programmer's Perspective
The Obama campaign has been praised for its innovative use of technology. What was the key to its success?
Purpose-Built Languages
While often breaking the rules of traditional language design, the growing ecosystem of purpose-built "little" languages is an essential part of systems development.
Code Spelunking Redux
Is this subject important enough to warrant two articles in five years? I believe it is.
Better Scripts, Better Games
Smarter, more powerful scripting languages will improve game performance while making gameplay development more efficient.
Scaling in Games & Virtual Worlds
Online games and virtual worlds have familiar scaling requirements, but don't be fooled: everything you know is wrong.
XML Fever
Don't let delusions about XML develop into a virulent strain of XML fever.
High Performance Web Sites
Want to make your Web site fly? Focus on front-end performance.
Improving Performance on the Internet
Given the Internet's bottlenecks, how can we build fast, scalable content-delivery systems?
Eventually Consistent
Building reliable distributed systems at a worldwide scale demands trade-offs?between consistency and availability.
Building Scalable Web Services
Build only what you really need.
Software Transactional Memory: Why Is It Only a Research Toy?
The promise of STM may likely be undermined by its overheads and workload applicabilities.
Parallel Programming with Transactional Memory
While sometimes even writing regular, single-threaded programs can be quite challenging, trying to split a program into multiple pieces that can be executed in parallel adds a whole dimension of additional problems. Drawing upon the transaction concept familiar to most programmers, transactional memory was designed to solve some of these problems and make parallel programming easier. Ulrich Drepper from Red Hat shows us how it's done.
Erlang for Concurrent Programming
What role can programming languages play in dealing with concurrency? One answer can be found in Erlang, a language designed for concurrency from the ground up.
Real-World Concurrency
In this look at how concurrency affects practitioners in the real world, Cantrill and Bonwick argue that much of the anxiety over concurrency is unwarranted.
The Five-Minute Rule 20 Years Later:
and How Flash Memory Changes the Rules
The old rule continues to evolve, while flash memory adds two new rules.
Enterprise SSDs
Solid-state drives are finally ready for the enterprise. But beware, not all SSDs are created alike.
Flash Storage Today
Can flash memory become the foundation for a new tier in the storage hierarchy?
Flash Disk Opportunity for Server Applications
Future flash-based disks could provide breakthroughs in IOPS, power, reliability, and volumetric capacity when compared with conventional disks.
A Pioneer's Flash of Insight
Jim Gray's vision of flash-based storage anchors this issue's theme.
Distributed Computing Economics
Computing economics are changing. Today there is rough price parity between: (1) one database access; (2) 10 bytes of network traffic; (3) 100,000 instructions; (4) 10 bytes of disk storage; and (5) a megabyte of disk bandwidth. This has implications for how one structures Internet-scale distributed computing: one puts computing as close to the data as possible in order to avoid expensive network traffic.
Ode to a Sailor
sailor, fleeting mood image of you; all sailor in bear grace, rough hands and poetic dream;
A Tribute to Jim Gray
Computer science attracts many very smart people, but a few stand out above the others, somehow blessed with a kind of creativity that most of us are denied. Names such as Alan Turing, Edsger Dijkstra, and John Backus come to mind. Jim Gray is another.
BASE: An Acid Alternative
In partitioned databases, trading some consistency for availability can lead to dramatic improvements in scalability.
Exposing the ORM Cache
Familiarity with ORM caching issues can help prevent performance problems and bugs.
ORM in Dynamic Languages
O/R mapping frameworks for dynamic languages such as Groovy provide a different flavor of ORM that can greatly simplify application code.
Bridging the Object-Relational Divide
ORM technologies can simplify data access, but be aware of the challenges that come with introducing this new layer of abstraction.
The Yin and Yang of Software Development
The C/C++ Solution Manager at Parasoft explains how infrastructure elements allow development teams to increase productivity without restricting creativity.
Managing Collaboration
Jeff Johnstone of TechExcel explains why there is a need for a new approach to application lifecycle management that better reflects the business requirements and challenges facing development teams.
Getting Bigger Reach Through Speech
Mark Ericson, vice president of product strategy for BlueNote Networks argues that in order to take advantage of new voice technologies you have to have a plan for integrating that capability directly into the applications that drive your existing business processes.
From Liability to Advantage: A Conversation with John Graham-Cumming and John Ousterhout
Software production has become a bottleneck in many development organizations.
Arm Your Applications for Bulletproof Deployment: A Conversation with Tom Spalthoff
Companies can achieve a reliable desktop environment while reducing the time and cost spent preparing high-quality application packages.
Intellectual Property and Software Piracy:
The Power of IP Protection and Software Licensing, an interview with Aladdin vice president Gregg Gronowski
The Power of IP Protection and Software Licensing, an interview with Aladdin vice president Gregg Gronowski
Reconfigurable Future
The Ability to Produce Cheaper, More Compact Chips is a Double-edged Sword.
The Emergence of iSCSI
Modern SCSI, as defined by the SCSI-3 Architecture Model, or SAM, really considers the cable and physical interconnections to storage as only one level in a larger hierarchy.
A New High-Performance Networked File System
This emerging file-access protocol dramatically enhances the flow of data over a network, making life easier in the data center.
Future Graphics Architectures
Graphics architectures are in the midst of a major transition. In the past, these were specialized architectures designed to support a single rendering algorithm: the standard Z buffer. Realtime 3D graphics has now advanced to the point where the Z-buffer algorithm has serious shortcomings for generating the next generation of higher-quality visual effects demanded by games and other interactive 3D applications. There is also a desire to use the high computational capability of graphics architectures to support collision detection, approximate physics simulations, scene management, and simple artificial intelligence.
Scalable Parallel Programming with CUDA
The advent of multicore CPUs and manycore GPUs means that mainstream processor chips are now parallel systems. Furthermore, their parallelism continues to scale with Moore's law. The challenge is to develop mainstream application software that transparently scales its parallelism to leverage the increasing number of processor cores, much as 3D graphics applications transparently scale their parallelism to manycore GPUs with widely varying numbers of cores.
Data-Parallel Computing
Users always care about performance. Although often it's just a matter of making sure the software is doing only what it should, there are many cases where it is vital to get down to the metal and leverage the fundamental characteristics of the processor.
GPUs: A Closer Look
A gamer wanders through a virtual world rendered in near- cinematic detail. Seconds later, the screen fills with a 3D explosion, the result of unseen enemies hiding in physically accurate shadows. Disappointed, the user exits the game and returns to a computer desktop that exhibits the stylish 3D look-and-feel of a modern window manager. Both of these visual experiences require hundreds of gigaflops of computing performance, a demand met by the GPU (graphics processing unit) present in every consumer PC.
How OSGi Changed My Life
In the early 1980s I discovered OOP (object-oriented programming) and fell in love with it, head over heels. As usual, this kind of love meant convincing management to invest in this new technology, and most important of all, send me to cool conferences. So I pitched the technology to my manager. I sketched him the rosy future, how one day we would create applications from ready-made classes. We would get those classes from a repository, put them together, and voila, a new application would be born.
Network Virtualization:
Breaking the Performance Barrier
The recent resurgence in popularity of virtualization has led to its use in a growing number of contexts, many of which require high-performance networking. Consider server consolidation, for example. The efficiency of network virtualization directly impacts the number of network servers that can effectively be consolidated onto a single physical machine. Unfortunately, modern network virtualization techniques incur significant overhead, which limits the achievable network performance. We need new network virtualization techniques to realize the full benefits of virtualization in network-intensive domains.
The Cost of Virtualization
Virtualization can be implemented in many different ways. It can be done with and without hardware support. The virtualized operating system can be expected to be changed in preparation for virtualization, or it can be expected to work unchanged. Regardless, software developers must strive to meet the three goals of virtualization spelled out by Gerald Popek and Robert Goldberg: fidelity, performance, and safety.
Beyond Server Consolidation
Virtualization technology was developed in the late 1960s to make more efficient use of hardware. Hardware was expensive, and there was not that much available.
Meet the Virts
When you dig into the details of supposedly overnight success stories, you frequently discover that they've actually been years in the making. Virtualization has been around for more than 30 years since the days when some of you were feeding stacks of punch cards into very physical machines yet in 2007 it tipped. VMware was the IPO sensation of the year; in November 2007 no fewer than four major operating system vendors (Microsoft, Oracle, Red Hat, and Sun) announced significant new virtualization capabilities; and among fashionable technologists it seems virtual has become the new black.
Big Games, Small Screens
One thing that becomes immediately apparent when creating and distributing mobile 3D games is that there are fundamental differences between the cellphone market and the more traditional games markets, such as consoles and handheld gaming devices. The most striking of these are the number of delivery platforms; the severe constraints of the devices, including small screens whose orientation can be changed; limited input controls; the need to deal with other tasks; the nonphysical delivery mechanism; and the variations in handset performance and input capability.
Understanding DRM
The explosive growth of the Internet and digital media has created both tremendous opportunities and new threats for content creators. Advances in digital technology offer new ways of marketing, disseminating, interacting with, and monetizing creative works, giving rise to expanding markets that did not exist just a few years ago. At the same time, however, the technologies have created major challenges for copyright holders seeking to control the distribution of their works and protect against piracy.
Document & Media Exploitation
A computer used by Al Qaeda ends up in the hands of a Wall Street Journal reporter. A laptop from Iran is discovered that contains details of that country's nuclear weapons program. Photographs and videos are downloaded from terrorist Web sites.
Powering Down
Power management - from laptops to rooms full of servers - is a topic of interest to everyone. In the beginning there was the desktop computer. It ran at a fixed speed and consumed less power than the monitor it was plugged into. Where computers were portable, their sheer size and weight meant that you were more likely to be limited by physical strength than battery life. It was not a great time for power management.
Storage Virtualization Gets Smart
Over the past 20 years we have seen the transformation of storage from a dumb resource with fixed reliability, performance, and capacity to a much smarter resource that can actually play a role in how data is managed. In spite of the increasing capabilities of storage systems, however, traditional storage management models have made it hard to leverage these data management capabilities effectively. The net result has been overprovisioning and underutilization. In short, although the promise was that smart shared storage would simplify data management, the reality has been different.
Hard Disk Drives:
The Good, the Bad and the Ugly!
HDDs (hard-disk drives) are like the bread in a peanut butter and jelly sandwich—sort of an unexciting piece of hardware necessary to hold the "software." They are simply a means to an end. HDD reliability, however, has always been a significant weak link, perhaps the weak link, in data storage. In the late 1980s people recognized that HDD reliability was inadequate for large data storage systems so redundancy was added at the system level with some brilliant software algorithms, and RAID (redundant array of inexpensive disks) became a reality. RAID moved the reliability requirements from the HDD itself to the system of data disks.
Standardizing Storage Clusters
Data-intensive applications such as data mining, movie animation, oil and gas exploration, and weather modeling generate and process huge amounts of data. File-data access throughput is critical for good performance. To scale well, these HPC (high-performance computing) applications distribute their computation among numerous client machines. HPC clusters can range from hundreds to thousands of clients with aggregate I/O demands ranging into the tens of gigabytes per second.
Voyage in the Agile Memeplex
Agile processes are not a technology, not a science, not a product. They constitute a space somewhat hard to define. Agile methods, or more precisely 'agile software development methods or processes', are a family of approaches and practices for developing software systems. Any attempt to define them runs into egos and marketing posturing.
Usablity Testing for the Web
Today's Internet user has more choices than ever before, with many competing sites offering similar services. This proliferation of options provides ample opportunity for users to explore different sites and find out which one best suits their needs for any particular service. Users are further served by the latest generation of Web technologies and services, commonly dubbed Web 2.0, which enables a better, more personalized user experience and encourages user-generated content.
Phishing Forbidden
Phishing is a significant risk facing Internet users today.1,2 Through e-mails or instant messages, users are led to counterfeit Web sites designed to trick them into divulging usernames, passwords, account numbers, and personal information. It is up to the user to ensure the authenticity of the Web site.
Building Secure Web Applications
In these days of phishing and near-daily announcements of identity theft via large-scale data losses, it seems almost ridiculous to talk about securing the Web. At this point most people seem ready to throw up their hands at the idea or to lock down one small component that they can control in order to keep the perceived chaos at bay.
Toward a Commodity Enterprise Middleware
Can AMQP enable a new era in messaging middleware? AMQP (Advanced Message Queuing Protocol) was born out of my own experience and frustrations in developing front- and back-office processing systems at investment banks. It seemed to me that we were living in integration Groundhog Day - the same problems of connecting systems together would crop up with depressing regularity. Each time the same discussions about which products to use would happen, and each time the architecture of some system would be curtailed to allow for the fact that the chosen middleware was reassuringly expensive.
The Seven Deadly Sins of Linux Security
Avoid these common security risks like the devil.
API Design Matters
Why changing APIs might become a criminal offense. After more than 25 years as a software engineer, I still find myself underestimating the time it will take to complete a particular programming task. Sometimes, the resulting schedule slip is caused by my own shortcomings: as I dig into a problem, I simply discover that it is a lot harder than I initially thought, so the problem takes longer to solvesuch is life as a programmer. Just as often I know exactly what I want to achieve and how to achieve it, but it still takes far longer than anticipated. When that happens, it is usually because I am struggling with an API that seems to do its level best to throw rocks in my path and make my life difficult.
Beyond Beowulf Clusters
In the early '90s, the Berkeley NOW (Network of Workstations) Project under David Culler posited that groups of less capable machines (running SunOS) could be used to solve scientific and other computing problems at a fraction of the cost of larger computers. In 1994, Donald Becker and Thomas Sterling worked to drive the costs even lower by adopting the then-fledgling Linux operating system to build Beowulf clusters at NASA's Goddard Space Flight Center. By tying desktop machines together with open source tools such as PVM (Parallel Virtual Machine), MPI (Message Passing Interface), and PBS (Portable Batch System), early clusters—which were often PC towers stacked on metal shelves with a nest of wires interconnecting them—fundamentally altered the balance of scientific computing.
The Evolution of Security
Security people are never in charge unless an acute embarrassment has occurred. Otherwise, their advice is tempered by "economic reality," which is to say that security is a means, not an end. This is as it should be. Since means are about tradeoffs, security is about trade-offs, but you knew all that.
DNS Complexity
DNS (domain name system) is a distributed, coherent, reliable, autonomous, hierarchical database, the first and only one of its kind. Created in the 1980s when the Internet was still young but overrunning its original system for translating host names into IP addresses, DNS is one of the foundation technologies that made the worldwide Internet (and the World Wide Web) possible. Yet this did not all happen smoothly, and DNS technology has been periodically refreshed and refined. Though it's still possible to describe DNS in simple terms, the underlying details are by now quite sublime.
Unified Communications with SIP
SIP can provide realtime communications as a network service.
Making SIP Make Cents
P2P payments using SIP could enable new classes of micropayment applications and business models.
Decentralizing SIP
If you're looking for a low-maintenance IP communications network, peer-to-peer SIP might be just the thing.
Basics and Beyond
More than just a simple telephony application protocol, SIP is a framework for developing communications systems.
Realtime Garbage Collection
It's now possible to develop realtime systems using Java.
Open vs. Closed:
Which Source is More Secure?
Which source is more secure?
One Step Ahead
Security vulnerabilities abound, but a few simple steps can minimize your risk.
Better, Faster, More Secure
Since I started a stint as chair of the IETF (Internet Engineering Task Force) in March 2005, I have frequently been asked, "What's coming next?" but I have usually declined to answer. Nobody is in charge of the Internet, which is a good thing, but it makes predictions difficult (and explains why this article starts with a disclaimer: It represents my views alone and not those of my colleagues at either IBM or the IETF).
The Virtualization Reality
A number of important challenges are associated with the deployment and configuration of contemporary computing infrastructure. Given the variety of operating systems and their many versions—including the often-specific configurations required to accommodate the wide range of popular applications—it has become quite a conundrum to establish and manage such systems.
Unlocking Concurrency
Multicore architectures are an inflection point in mainstream software development because they force developers to write parallel programs. In a previous article in Queue, Herb Sutter and James Larus pointed out, "The concurrency revolution is primarily a software revolution.
Playing for Keeps
Inflection points come at you without warning and quickly recede out of reach. We may be nearing one now. If so, we are now about to play for keeps, and "we" doesn't mean just us security geeks. If anything, it's because we security geeks have not worked the necessary miracles already that an inflection point seems to be approaching at high velocity.
Criminal Code:
The Making of a Cybercriminal
NOTE: This is a fictional account of malware creators and their experiences. Although the characters are made up, the techniques and events are patterned on real activities of many different groups developing malicious software.
E-mail Authentication:
What, Why, How?
Internet e-mail was conceived in a different world than we live in today. It was a small, tightly knit community, and we didn't really have to worry too much about miscreants. Generally, if someone did something wrong, the problem could be dealt with through social means; "shunning" is very effective in small communities.
An Epidemic
Painted in the broadest of strokes, cybercrime essentially is the leveraging of information systems and technology to commit larceny, extortion, identity theft, fraud, and, in some cases, corporate espionage. Who are the miscreants who commit these crimes, and what are their motivations? One might imagine they are not the same individuals committing crimes in the physical world. Bank robbers and scam artists garner a certain public notoriety after only a few occurrences of their crimes, yet cybercriminals largely remain invisible and unheralded. Based on sketchy news accounts and a few public arrests, such as Mafiaboy, accused of paralyzing Amazon, CNN, and other Web sites, the public may infer these miscreants are merely a subculture of teenagers.
Breaking the Major Release Habit
Can agile development make your team more productive?
The Heart of Eclipse
A look inside and extensible plug-in architecture ECLIPSE is both an open, extensible development environment for building software and an open, extensible application framework upon which software can be built. Considered the most popular Java IDE, it provides a common UI model for working with tools and promotes rapid development of modular features based on a plug-in component model. The Eclipse Foundation designed the platform to run natively on multiple operating systems, including Macintosh, Windows, and Linux, providing robust integration with each and providing rich clients that support the GUI interactions everyone is familiar with: drag and drop, cut and paste (clipboard), navigation, and customization.
The Long Road to 64 Bits
"Double, double, toil and trouble"... Shakespeare's words (Macbeth, Act 4, Scene 1) often cover circumstances beyond his wildest dreams. Toil and trouble accompany major computing transitions, even when people plan ahead. To calibrate "tomorrow's legacy today," we should study "tomorrow's legacy yesterday." Much of tomorrow's software will still be driven by decades-old decisions. Past decisions have unanticipated side effects that last decades and can be difficult to undo.
Keeping Score in the IT Compliance Game
Achieving developer acceptance of standardized procedures for managing applications from development to release is one of the largest hurdles facing organizations today. Establishing a standardized development-to-release workflow, often referred to as the ALM (application lifecycle management) process, is particularly critical for organizations in their efforts to meet tough IT compliance mandates. This is much easier said than done, as different development teams have created their own unique procedures that are undocumented, unclear, and nontraceable.
Compliance Deconstructed
The topic of compliance becomes increasingly complex each year. Dozens of regulatory requirements can affect a company's business processes. Moreover, these requirements are often vague and confusing. When those in charge of compliance are asked if their business processes are in compliance, it is understandably difficult for them to respond succinctly and with confidence. This article looks at how companies can deconstruct compliance, dealing with it in a systematic fashion and applying technology to automate compliance-related business processes. It also looks specifically at how Microsoft approaches compliance to SOX (Sarbanes-Oxley Act of 2002).
Box Their SOXes Off
Data is a precious resource for any large organization. The larger the organization, the more likely it will rely to some degree on third-party vendors and partners to help it manage and monitor its mission-critical data. In the wake of new regulations for public companies, such as Section 404 of SOX (Sarbanes-Oxley Act of 2002), the folks who run IT departments for Fortune 1000 companies have an ever-increasing need to know that when it comes to the 24/7/365 monitoring of their critical data transactions, they have business partners with well-planned and well-documented procedures.
Complying with Compliance
"Hey, compliance is boring. Really, really boring. And besides, I work neither in the financial industry nor in health care. Why should I care about SOX and HIPAA?" Yep, you're absolutely right. You write payroll applications, or operating systems, or user interfaces, or (heaven forbid) e-mail servers. Why should you worry about compliance issues?
A Requirements Primer
Many software engineers and architects are exposed to compliance through the growing number of rules, regulations, and standards with which their employers must comply. Some of these requirements, such as HIPAA (Health Insurance Portabililty and Accountability Act), focus primarily on one industry, whereas others, such as SOX (Sarbanes-Oxley Act), span many industries. Some apply to only one country, while others cross national boundaries. To help navigate this often confusing world, Queue has assembled a short primer that provides background on four of the most important compliance challenges that organizations face today.
Too Much Information
Two applications reveal the key challenges in making context-aware computing a reality. As mobile computing devices and a variety of sensors become ubiquitous, new resources for applications and services - often collectively referred to under the rubric of context-aware computing - are becoming available to designers and developers. In this article, we consider the potential benefits and issues that arise from leveraging context awareness in new communication services that include the convergence of VoIP (voice over IP) and traditional information technology.
The Invisible Assistant
One lab's experiment with ubiquitous computing
Social Perception
Modeling human interaction for the next generation of communication services
The Future of Human-Computer Interaction
Personal computing launched with the IBM PC. But popular computing launched with the modern WIMP (windows, icons, mouse, pointer) interface, which made computers usable by ordinary people.
The Integration Challenge
The promise of software as a service is becoming a reality with many ASPs (application service providers). Organizations using ASPs and third-party vendors that provide value-added products to ASPs need to integrate with them. ASPs enable this integration by providing Web service-based APIs. There are significant differences between integrating with ASPs over the Internet and integrating with a local application. When integrating with ASPs, users have to consider a number of issues, including latency, unavailability, upgrades, performance, load limiting, and lack of transaction support.
Untangling Enterprise Java
Separation of concerns is one of the oldest concepts in computer science. The term was coined by Dijkstra in 1974.1 It is important because it simplifies software, making it easier to develop and maintain. Separation of concerns is commonly achieved by decomposing an application into components. There are, however, crosscutting concerns, which span (or cut across) multiple components. These kinds of concerns cannot be handled by traditional forms of modularization and can make the application more complex and difficult to maintain.
The Rise and Fall of CORBA
Depending on exactly when one starts counting, CORBA is about 10-15 years old. During its lifetime, CORBA has moved from being a bleeding-edge technology for early adopters, to being a popular middleware, to being a niche technology that exists in relative obscurity. It is instructive to examine why CORBA—despite once being heralded as the "next-generation technology for e-commerce"—suffered this fate. CORBA's history is one that the computing industry has seen many times, and it seems likely that current middleware efforts, specifically Web services, will reenact a similar history.
From COM to Common
Ten years ago, the term component software meant something relatively specific and concrete. A small number of software component frameworks more or less defined the concept for most people. Today, few terms in the software industry are less precise than component software. There are now many different forms of software componentry for many different purposes. The technologies and methodologies of 10 years ago have evolved in fundamental ways and have been joined by an explosion of new technologies and approaches that have redefined our previously held notions of component software.
The Network's New Role
Application-oriented networks can help bridge the gap between enterprises.
Search Considered Integral
A combination of tagging, categorization, and navigation can help end-users leverage the power of enterprise search.
AI Gets a Brain
New technology allows software to tap real human intelligence.
Java in a Teacup
Programming Bluetooth-enabled devices using J2ME
The challenges of delivering a reliable, easy-to-use DVR service to the masses
The (not so) Hidden Computer
The growing complexity of purpose-built systems is making it difficult to conceal the computers within.
Under New Management
Autonomic computing is revolutionizing the way we manage complex systems.
Best Practice (BPM)
In business process management, finding the right tool suite is just the beginning.
People and Process
Minimizing the pain of business process change
Going with the Flow
Workflow systems can provide value beyond automating business processes.
Modern Performance Monitoring
Today's diverse and decentralized computer world demands new thinking about performance monitoring and analysis.
Performance Anti-Patterns
Want your apps to run faster? Here's what not to do.
A High-Performance Team
From design to production, performance should be part of the process.
Hidden in Plain Sight
Improvements in the observability of software can help you diagnose your most crippling performance problems.
Coding for the Code
Can models provide the DNA for software development?
Monitoring, at Your Service
Automated monitoring can increase the reliability and scalability of today's online software services.
Lessons from the Floor
The manufacturing industry can teach us a lot about measuring performance in large-scale Internet services.
Information Extraction:
Distilling Structured Data from Unstructured Text
Distilling structured data from unstructured text
Threads without the Pain
Multithreaded programming need not be so angst-ridden.
Fighting Spam with Reputation Systems
User-submitted spam fingerprints
Social Bookmarking in the Enterprise
Can your organization benefit from social bookmarking tools?
Why Your Data Won't Mix
When independent parties develop database schemas for the same domain, they will almost always be quite different from each other. These differences are referred to as semantic heterogeneity, which also appears in the presence of multiple XML documents, Web services, and ontologies—or more broadly, whenever there is more than one way to structure a body of data. The presence of semi-structured data exacerbates semantic heterogeneity, because semi-structured schemas are much more flexible to start with. For multiple data systems to cooperate with each other, they must understand each other's schemas.
Order from Chaos
There is probably little argument that the past decade has brought the "big bang" in the amount of online information available for processing by humans and machines. Two of the trends that it spurred (among many others) are: first, there has been a move to more flexible and fluid (semi-structured) models than the traditional centralized relational databases that stored most of the electronic data before; second, today there is simply too much information available to be processed by humans, and we really need help from machines.
XML <and Semi-Structured Data>
XML, as defined by the World Wide Web Consortium in 1998, is a method of marking up a document or character stream to identify structural or other units within the data. XML makes several contributions to solving the problem of semi-structured data, the term database theorists use to denote data that exhibits any of the following characteristics:
Learning from the Web
In the past decade we have seen a revolution in computing that transcends anything seen to date in terms of scope and reach, but also in terms of how we think about what makes up "good" and "bad" computing. The Web taught us several unintuitive lessons:
Managing Semi-Structured Data
I vividly remember during my first college class my fascination with the relational database—an information oasis that guaranteed a constant flow of correct, complete, and consistent information at our disposal. In that class I learned how to build a schema for my information, and I learned that to obtain an accurate schema there must be a priori knowledge of the structure and properties of the information to be modeled.
Software and the Concurrency Revolution
Leveraging the full power of multicore processors demands new tools and new thinking from the software industry. Concurrency has long been touted as the "next big thing" and "the way of the future," but for the past 30 years, mainstream software development has been able to ignore it. Our parallel future has finally arrived: new machines will be parallel machines, and this will require major changes in the way we develop software. The introductory article in this issue ("The Future of Microprocessors" by Kunle Olukotun and Lance Hammond) describes the hardware imperatives behind this shift in computer architecture from uniprocessors to multicore processors, also known as CMPs (chip multiprocessors).
The Price of Performance
In the late 1990s, our research group at DEC was one of a growing number of teams advocating the CMP (chip multiprocessor) as an alternative to highly complex single-threaded CPUs. We were designing the Piranha system,1 which was a radical point in the CMP design space in that we used very simple cores (similar to the early RISC designs of the late '80s) to provide a higher level of thread-level parallelism. Our main goal was to achieve the best commercial workload performance for a given silicon budget.
Extreme Software Scaling
The advent of SMP (symmetric multiprocessing) added a new degree of scalability to computer systems. Rather than deriving additional performance from an incrementally faster microprocessor, an SMP system leverages multiple processors to obtain large gains in total system performance. Parallelism in software allows multiple jobs to execute concurrently on the system, increasing system throughput accordingly. Given sufficient software parallelism, these systems have proved to scale to several hundred processors.
The Future of Microprocessors
The performance of microprocessors that power modern computers has continued to increase exponentially over the years for two main reasons. First, the transistors that are the heart of the circuits in all processors and memory chips have simply become faster over time on a course described by Moore's law,1 and this directly affects the performance of processors built with those transistors. Moreover, actual processor performance has increased faster than Moore's law would predict,2 because processor designers have been able to harness the increasing numbers of transistors available on modern chips to extract more parallelism from software.
Enterprise Grid Computing
I have to admit a great measure of sympathy for the IT populace at large, when it is confronted by the barrage of hype around grid technology, particularly within the enterprise. Individual vendors have attempted to plant their flags in the notionally virgin technological territory and proclaim it as their own, using terms such as grid, autonomic, self-healing, self-managing, adaptive, utility, and so forth. Analysts, well, analyze and try to make sense of it all, and in the process each independently creates his or her own map of this terra incognita, naming it policy-based computing, organic computing, and so on.
Web Services and IT Management
Web services aren't just for application integration anymore.
Enterprise Software as Service
While the practice of outsourcing business functions such as payroll has been around for decades, its realization as online software services has only recently become popular. In the online service model, a provider develops an application and operates the servers that host it. Customers access the application over the Internet using industry-standard browsers or Web services clients. A wide range of online applications, including e-mail, human resources, business analytics, CRM (customer relationship management), and ERP (enterprise resource planning), are available.
Describing the Elephant:
The Different Faces of IT as Service
In a well-known fable, a group of blind men are asked to describe an elephant. Each encounters a different part of the animal and, not surprisingly, provides a different description. We see a similar degree of confusion in the IT industry today, as terms such as service-oriented architecture, grid, utility computing, on-demand, adaptive enterprise, data center automation, and virtualization are bandied about. As when listening to the blind men, it can be difficult to know what reality lies behind the words, whether and how the different pieces fit together, and what we should be doing about the animal(s) that are being described.
Programmers Are People, too
I would like to start out this article with an odd, yet surprisingly uncontroversial assertion, which is this: programmers are human. I wish to use this as a premise to explore how to improve the programmer's lot. So, please, no matter your opinion on the subject, grant me this assumption for the sake of argument.
Attack Trends:
2004 and 2005
Counterpane Internet Security Inc. monitors more than 450 networks in 35 countries, in every time zone. In 2004 we saw 523 billion network events, and our analysts investigated 648,000 security "tickets." What follows is an overview of what's happening on the Internet right now, and what we expect to happen in the coming months.
Security - Problem Solved?
There are plenty of security problems that have solutions. Yet, our security problems don't seem to be going away. What's wrong here? Are consumers being offered snake oil and rejecting it? Are they not adopting solutions they should be adopting? Or, is there something else at work, entirely? We'll look at a few places where the world could easily be a better place, but isn't, and build some insight as to why.
The Answer is 42 of Course
Why is security so hard? As a security consultant, I'm glad that people feel that way, because that perception pays my mortgage. But is it really so difficult to build systems that are impenetrable to the bad guys?
You Don't Know Jack about Network Performance
Why does an application that works just fine over a LAN come to a grinding halt across the wide-area network? You may have experienced this firsthand when trying to open a document from a remote file share or remotely logging in over a VPN to an application running in headquarters. Why is it that an application that works fine in your office can become virtually useless over the WAN? If you think it's simply because there's not enough bandwidth in the WAN, then you don't know jack about network performance.
Streams and Standards:
Delivering Mobile Video
Don't believe me? Follow along… Mobile phones are everywhere. Everybody has one. Think about the last time you were on an airplane and the flight was delayed on the ground. Immediately after the dreaded announcement, you heard everyone reach for their phones and start dialing.
Mobile Media:
Making It a Reality
Many future mobile applications are predicated on the existence of rich, interactive media services. The promise and challenge of such services is to provide applications under the most hostile conditions - and at low cost to a user community that has high expectations. Context-aware services require information about who, where, when, and what a user is doing and must be delivered in a timely manner with minimum latency. This article reveals some of the current state-of-the-art "magic" and the research challenges.
Enterprise-Grade Wireless
We have been working in the wireless space in one form or another in excess of 10 years and have participated in every phase of its maturation process. We saw wireless progress from a toy technology before the dot-com boom, to something truly promising during the boom, only to be left wanting after the bubble when the technology was found to be not ready for prime time. Fortunately, it appears that we have finally reached the point where the technology and the enterprise's expectations have finally converged.
Beyond Relational Databases
There is more to data access than SQL.
Databases of Discovery
Open-ended database ecosystems promote new discoveries in biotech. Can they help your organization, too?
A Call to Arms
Long anticipated, the arrival of radically restructured database architectures is now finally at hand.
UML Fever:
Diagnosis and Recovery
Acknowledgment is only the first step toward recovery from this potentially devastating affliction. The Institute of Infectious Diseases has recently published research confirming that the many and varied strains of UML Fever continue to spread worldwide, indiscriminately infecting software analysts, engineers, and managers alike. One of the fevers most serious side effects has been observed to be a significant increase in both the cost and duration of developing software products. This increase is largely attributable to a decrease in productivity resulting from fever-stricken individuals investing time and effort in activities that are of little or no value to producing deliverable products.
On Plug-ins and Extensible Architectures
Extensible application architectures such as Eclipse offer many advantages, but one must be careful to avoid "plug-in hell."
Patching the Enterprise
Organizations of all sizes are spending considerable efforts on getting patch management right - their businesses depend on it.
Understanding Software Patching
Developing and deploying patches is an increasingly important part of the software development process.
A Passage to India
Most American IT employees take a dim view of offshore outsourcing. It's considered unpatriotic and it drains valuable intellectual capital and jobs from the United States to destinations such as India or China. Online discussion forums on sites such as isyourjobgoingoffshore.com are headlined with titles such as "How will you cope?" and "Is your career in danger?" A cover story in BusinessWeek magazine a couple of years ago summed up the angst most people suffer when faced with offshoring: "Is your job next?"
Orchestrating an Automated Test Lab
Networking and the Internet are encouraging increasing levels of interaction and collaboration between people and their software. Whether users are playing games or composing legal documents, their applications need to manage the complex interleaving of actions from multiple machines over potentially unreliable connections. As an example, Silicon Chalk is a distributed application designed to enhance the in-class experience of instructors and students. Its distributed nature requires that we test with multiple machines. Manual testing is too tedious, expensive, and inconsistent to be effective. While automating our testing, however, we have found it very labor intensive to maintain a set of scripts describing each machine's portion of a given test.
Sifting Through the Software Sandbox:
SCM Meets QA
Thanks to modern SCM (software configuration management) systems, when developers work on a codeline they leave behind a trail of clues that can reveal what parts of the code have been modified, when, how, and by whom. From the perspective of QA (quality assurance) and test engineers, is this all just "data," or is there useful information that can improve the test coverage and overall quality of a product?
Too Darned Big to Test
The increasing size and complexity of software, coupled with concurrency and distributed systems, has made apparent the ineffectiveness of using only handcrafted tests. The misuse of code coverage and avoidance of random testing has exacerbated the problem. We must start again, beginning with good design (including dependency analysis), good static checking (including model property checking), and good unit testing (including good input selection). Code coverage can help select and prioritize tests to make you more efficient, as can the all-pairs technique for controlling the number of configurations.
Quality Assurance:
Much More than Testing
Quality assurance isn't just testing, or analysis, or wishful thinking. Although it can be boring, difficult, and tedious, QA is nonetheless essential.
Self-Healing in Modern Operating Systems
A few early steps show there's a long (and bumpy) road ahead.
How Not to Write Fortran in Any Language
There are characteristics of good coding that transcend all programming languages.
Extensible Programming for the 21st Century
Is an open, more flexible programming environment just around the corner?
Fuzzy Boundaries:
Objects, Components, and Web Services
It's easy to transform objects into components and Web services, but how do we know which is right for the job?
Languages, Levels, Libraries, and Longevity
New programming languages are born every day. Why do some succeed and some fail? In 50 years, we've already seen numerous programming systems come and (mostly) go, although some have remained a long time and will probably do so for: decades? centuries? millennia? The questions about language designs, levels of abstraction, libraries, and resulting longevity are numerous. Why do new languages arise? Why is it sometimes easier to write new software than to adapt old software that works? How many different levels of languages make sense? Why do some languages last in the face of "better" ones?
Lack of Priority Queuing Considered Harmful
Most modern routers consist of several line cards that perform packet lookup and forwarding, all controlled by a control plane that acts as the brain of the router, performing essential tasks such as management functions, error reporting, control functions including route calculations, and adjacency maintenance. This control plane has many names; in this article it is the route processor, or RP. The route processor calculates the forwarding table and downloads it to the line cards using a control-plane bus. The line cards perform the actual packet lookup and forwarding.
Outsourcing: Devising a Game Plan
Your CIO just summoned you to duty by handing off the decision-making power about whether to outsource next years big development project to rewrite the internal billing system. That's quite a daunting task! How can you possibly begin to decide if outsourcing is the right option for your company? There are a few strategies that you can follow to help you avoid the pitfalls of outsourcing and make informed decisions. Outsourcing is not exclusively a technical issue, but it is a decision that architects or development managers are often best qualified to make because they are in the best position to know what technologies make sense to keep in-house.
Error Messages:
What's the Problem?
Computer users spend a lot of time chasing down errors - following the trail of clues that starts with an error message and that sometimes leads to a solution and sometimes to frustration. Problems with error messages are particularly acute for system administrators (sysadmins) - those who configure, install, manage, and maintain the computational infrastructure of the modern world - as they spend a lot of effort to keep computers running amid errors and failures.
Automating Software Failure Reporting
We can only fix those bugs we know about.
Oops! Coping with Human Error in IT Systems
Errors Happen. How to Deal.
Trials and Tribulations of Debugging Concurrency
We now sit firmly in the 21st century where the grand challenge to the modern-day programmer is neither memory leaks nor type issues (both of those problems are now effectively solved), but rather issues of concurrency. How does one write increasingly complex programs where concurrency is a first-class concern. Or even more treacherous, how does one debug such a beast? These questions bring fear into the hearts of even the best programmers.
Thread Scheduling in FreeBSD 5.2
A busy system makes thousands of scheduling decisions per second, so the speed with which scheduling decisions are made is critical to the performance of the system as a whole. This article - excerpted from the forthcoming book, "The Design and Implementation of the FreeBSD Operating System" - uses the example of the open source FreeBSD system to help us understand thread scheduling. The original FreeBSD scheduler was designed in the 1980s for large uniprocessor systems. Although it continues to work well in that environment today, the new ULE scheduler was designed specifically to optimize multiprocessor and multithread environments. This article first studies the original FreeBSD scheduler, then describes the new ULE scheduler.
Integrating RFID
RFID (radio frequency identification) has received a great deal of attention in the commercial world over the past couple of years. The excitement stems from a confluence of events. First, through the efforts of the former Auto-ID Center and its sponsor companies, the prospects of low-cost RFID tags and a networked supply chain have come within reach of a number of companies. Second, several commercial companies and government bodies, such as Wal-Mart and Target in the United States, Tesco in Europe, and the U.S. Department of Defense, have announced RFID initiatives in response to technology improvements.
The Magic of RFID
Many modern technologies give the impression they work by magic, particularly when they operate automatically and their mechanisms are invisible. A technology called RFID (radio frequency identification), which is relatively new to the mass market, has exactly this characteristic and for many people seems a lot like magic. RFID is an electronic tagging technology that allows an object, place, or person to be automatically identified at a distance without a direct line-of-sight, using an electromagnetic challenge/response exchange.
A Time and a Place for Standards
History shows how abuses of the standards process have impeded progress. Over the next decade, we will encounter at least three major opportunities where success will hinge largely on our ability to define appropriate standards. That's because intelligently crafted standards that surface at just the right time can do much to nurture nascent industries and encourage product development simply by creating a trusted and reliable basis for interoperability. From where I stand, the three specific areas I see as particularly promising are: (1) all telecommunications and computing capabilities that work together to facilitate collaborative work; (2) hybrid computing/home entertainment products providing for the online distribution of audio and/or video content; and (3) wireless sensor and network platforms (the sort that some hope the 802.15.4 and ZigBee Alliance standards will ultimately enable).
VoIP Security: Not an Afterthought
Voice over IP (VoIP) promises to up-end a century-old model of voice telephony by breaking the traditional monolithic service model of the public switched telephone network (PSTN) and changing the point of control and provision from the central office switch to the end user's device.
VoIP: What is it Good for?
VoIP (voice over IP) technology is a rapidly expanding field. More and more VoIP components are being developed, while existing VoIP technology is being deployed at a rapid and still increasing pace. This growth is fueled by two goals: decreasing costs and increasing revenues.
Not Your Father's PBX?
Perhaps no piece of office equipment is more taken for granted than the common business telephone. The technology behind this basic communication device, however, is in the midst of a major transformation. Businesses are now converging their voice and data networks in order to simplify their network operations and take advantage of the new functional benefits and capabilities that a converged network delivers from greater productivity and cost savings to enhanced mobility.
You Don't Know Jack About VoIP
Telecommunications worldwide has experienced a significant revolution over recent years. The long-held promise of network convergence is occurring at an increasing pace. This convergence of data, voice, and video using IP-based networks is delivering advanced services at lower cost across the spectrum, including residential users, business customers of varying sizes, and service providers.
Leveraging Application Frameworks
In today's competitive, fast-paced computing industry, successful software must increasingly be: (1) extensible to support successions of quick updates and additions to address new requirements and take advantage of emerging markets; (2) flexible to support a growing range of multimedia data types, traffic flows, and end-to-end QoS (quality of service) requirements; (3) portable to reduce the effort required to support applications on heterogeneous operating-system platforms and compilers; (4) reliable to ensure that applications are robust and tolerant to faults; (5) scalable to enable applications to handle larger numbers of clients simultaneously; and (6) affordable to ensure that the total ownership costs of software acquisition and evolution are not prohibitively high.
Security is Harder than You Think
Many developers see buffer overflows as the biggest security threat to software and believe that there is a simple two-step process to secure software: switch from C or C++ to Java, then start using SSL (Secure Sockets Layer) to protect data communications. It turns out that this naïve tactic isn't sufficient. In this article, we explore why software security is harder than people expect, focusing on the example of SSL.
Virtual Machines of the Past (and Future)
Simulators are a form of "virtual machine" intended to address a simple problem: the absence of real hardware. Simulators for past systems address the loss of real hardware and preserve the usability of software after real hardware has vanished. Simulators for future systems address the variability of future hardware designs and facilitate the development of software before real hardware exists.
Building Systems to Be Shared, Securely
Want to securely partition VMs? One option is to put 'em in Jail.
Building Systems to Be Shared, Securely
Want to securely partition VMs? One option is to put 'em in Jail.
The Reincarnation of Virtual Machines
Virtualization makes a comeback.
The Hitchhiker's Guide to Biomorphic Software
The natural world may be the inspiration we need for solving our computer problems. While it is certainly true that "the map is not the territory," most visitors to a foreign country do prefer to take with them at least a guidebook to help locate themselves as they begin their explorations. That is the intent of this article. Although there will not be enough time to visit all the major tourist sites, with a little effort and using the information in the article as signposts, the intrepid explorer can easily find numerous other, interesting paths to explore.
Perfect Storm:
The Insider, Naivety, and Hostility
Every year corporations and government installations spend millions of dollars fortifying their network infrastructures. Firewalls, intrusion detection systems, and antivirus products stand guard at network boundaries, and individuals monitor countless logs and sensors for even the subtlest hints of network penetration. Vendors and IT managers have focused on keeping the wily hacker outside the network perimeter, but very few technological measures exist to guard against insiders - those entities that operate inside the fortified network boundary. The 2002 CSI/FBI survey estimates that 70 percent of successful attacks come from the inside. Several other estimates place those numbers even higher.
Network Forensics
The dictionary defines forensics as "the use of science and technology to investigate and establish facts in criminal or civil courts of law." I am more interested, however, in the usage common in the computer world: using evidence remaining after an attack on a computer to determine how the attack was carried out and what the attacker did.
Security: The Root of the Problem
Security bug? My programming language made me do it! It doesn't seem that a day goes by without someone announcing a critical flaw in some crucial piece of software or other. Is software that bad? Are programmers so inept? What the heck is going on, and why is the problem getting worse instead of better? One distressing aspect of software security is that we fundamentally don't seem to "get it."
Blaster Revisited
What lessons can we learn from the carnage the Blaster worm created? The following tale is based upon actual circumstances from corporate enterprises that were faced with confronting and eradicating the Blaster worm, which hit in August 2003. The story provides views from many perspectives, illustrating the complexity and sophistication needed to combat new blended threats.
From IR to Search, and Beyond
Searching has come a long way since the 60s, but have we only just begun?
TCP Offload to the Rescue
In recent years, TCP/IP offload engines, known as TOEs, have attracted a good deal of industry attention and a sizable share of venture capital dollars. A TOE is a specialized network device that implements a significant portion of the TCP/IP protocol in hardware, thereby offloading TCP/IP processing from software running on a general-purpose CPU. This article examines the reasons behind the interest in TOEs and looks at challenges involved in their implementation and deployment.
Desktop Linux: Where Art Thou?
Linux on the desktop has come a long way - and it's been a roller-coaster ride. At the height of the dot-com boom, around the time of Red Hat's initial public offering, people expected Linux to take off on the desktop in short order. A few years later, after the stock market crash and the failure of a couple of high-profile Linux companies, pundits were quick to proclaim the stillborn death of Linux on the desktop.
There's No Such Thing as a Free (Software) Lunch
"The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software to make sure the software is free for all its users." So begins the GNU General Public License, or GPL, which has become the most widely used of open source software licenses. Freedom is the watchword; it's no coincidence that the organization that wrote the GPL is called the Free Software Foundation and that open source developers everywhere proclaim, "Information wants to be free."
Is Open Source Right for You?:
A Fictional Case Study of Open Source in a Commercial Software Shop
The media often present open source software as a direct competitor to commercial software. This depiction, usually pitting David (Linux) against Goliath (Microsoft), makes for fun reading in the weekend paper. However, it mostly misses the point of what open source means to a development organization. In this article, I use the experiences of GizmoSoft (a fictitious software company) to present some perspectives on the impact of open source software usage in a software development shop.
Open Source to the Core
The open source development model is not exactly new. Individual engineers have been using open source as a collaborative development methodology for decades. Now that it has come to the attention of upper and middle management, however, it's finally being openly acknowledged as a commercial engineering force-multiplier and important option for avoiding significant software development costs.
Instant Messaging or Instant Headache?
It's a reality. You have IM (instant messaging) clients in your environment. You have already recognized that it is eating up more and more of your network bandwidth and with Microsoft building IM capability into its XP operating system and applications, you know this will only get worse. Management is also voicing concerns over the lost user productivity caused by personal conversations over this medium. You have tried blocking these conduits for conversation, but it is a constant battle.
Gaming Graphics:
The Road to Revolution
It has been a long journey from the days of multicolored sprites on tiled block backgrounds to the immersive 3D environments of modern games. What used to be a job for a single game creator is now a multifaceted production involving staff from every creative discipline. The next generation of console and home computer hardware is going to bring a revolutionary leap in available computing power; a teraflop (trillion floating-point operations per second) or more will be on tap from commodity hardware.
Building Nutch:
Open Source Search
Search engines are as critical to Internet use as any other part of the network infrastructure, but they differ from other components in two important ways. First, their internal workings are secret, unlike, say, the workings of the DNS (domain name system). Second, they hold political and cultural power, as users increasingly rely on them to navigate online content.
Why Writing Your Own Search Engine Is Hard
There must be 4,000 programmers typing away in their basements trying to build the next "world's most scalable" search engine. It has been done only a few times. It has never been done by a big group; always one to four people did the core work, and the big team came on to build the elaborations and the production infrastructure. Why is it so hard? We are going to delve a bit into the various issues to consider when writing a search engine. This article is aimed at those individuals or small groups that are considering this endeavor for their Web site or intranet.
Enterprise Search: Tough Stuff
The last decade has witnessed the growth of information retrieval from a boutique discipline in information and library science to an everyday experience for billions of people around the world. This revolution has been driven in large measure by the Internet, with vendors focused on search and navigation of Web resources and Web content management. Simultaneously, enterprises have invested in networking all of their information together to the point where it is increasingly possible for employees to have a single window into the enterprise.
Searching vs. Finding
Finding information and organizing it so that it can be found are two key aspects of any company's knowledge management strategy. Nearly everyone is familiar with the experience of searching with a Web search engine and using a search interface to search a particular Web site once you get there. (You may have even noticed that the latter often doesn't work as well as the former.) After you have a list of hits, you typically spend a significant amount of time following links, waiting for pages to download, reading through a page to see if it has what you want, deciding that it doesn't, backing up to try another link, deciding to try another way to phrase your request, et cetera.
BPM: The Promise and the Challenge
Over the last decade, businesses and governments have been giving increasing attention to business processes - to their description, automation, and management. This interest grows out of the need to streamline business operations, consolidate organizations, and save costs, reflecting the fact that the process is the basic unit of business value within an organization.
Death by UML Fever
A potentially deadly illness, clinically referred to as UML (Unified Modeling Language) fever, is plaguing many software-engineering efforts today. This fever has many different strains that vary in levels of lethality and contagion. A number of these strains are symptomatically related, however. Rigorous laboratory analysis has revealed that each is unique in origin and makeup. A particularly insidious characteristic of UML fever, common to most of its assorted strains, is the difficulty individuals and organizations have in self-diagnosing the affliction. A consequence is that many cases of the fever go untreated and often evolve into more complex and lethal strains.
Digitally Assisted Analog Integrated Circuits
In past decades, "Moore's law" has governed the revolution in microelectronics. Through continuous advancements in device and fabrication technology, the industry has maintained exponential progress rates in transistor miniaturization and integration density. As a result, microchips have become cheaper, faster, more complex, and more power efficient.
Stream Processors: Progammability and Efficiency
Many signal processing applications require both efficiency and programmability. Baseband signal processing in 3G cellular base stations, for example, requires hundreds of GOPS (giga, or billions, of operations per second) with a power budget of a few watts, an efficiency of about 100 GOPS/W (GOPS per watt), or 10 pJ/op (picoJoules per operation). At the same time programmability is needed to follow evolving standards, to support multiple air interfaces, and to dynamically provision processing resources over different air interfaces. Digital television, surveillance video processing, automated optical inspection, and mobile cameras, camcorders, and 3G cellular handsets have similar needs.
DSPs: Back to the Future
From the dawn of the DSP (digital signal processor), an old quote still echoes: "Oh, no! We'll have to use state-of-the-art 5µm NMOS!" The speaker's name is lost in the fog of history, as are many things from the ancient days of 5µm chip design. This quote refers to the first Bell Labs DSP whose mask set in fact underwent a 10 percent linear lithographic shrink to 4.5µm NMOS (N-channel metal oxide semiconductor) channel length and taped out in late 1979 with an aggressive full-custom circuit design.
On Mapping Alogrithms to DSP Architectures
Our complex world is characterized by representation, transmission, and storage of information - and information is mostly processed in digital form. With the advent of DSPs (digital signal processors), engineers are able to implement complex algorithms with relative ease. Today we find DSPs all around us - in cars, digital cameras, MP3 and DVD players, modems, and so forth. Their widespread use and deployment in complex systems has triggered a revolution in DSP architectures, which in turn has enabled engineers to implement algorithms of ever-increasing complexity.
Of Processors and Processing
Digital signal processing is a stealth technology. It is the core enabling technology in everything from your cellphone to the Mars Rover. It goes much further than just enabling a one-time breakthrough product. It provides ever-increasing capability; compare the performance gains made by dial-up modems with the recent performance gains of DSL and cable modems. Remarkably, digital signal processing has become ubiquitous with little fanfare, and most of its users are not even aware of what it is.
People in Our Software
People are not well represented in today's software. With the exception of IM (instant messaging) clients, today's applications offer few clues that people are actually living beings. Static strings depict things associated with people like e-mail addresses, phone numbers, and home-page URLs. Applications also tend to show the same information about a person, no matter who is viewing it.
Sensible Authentication
The problem with securing assets and their functionality is that, by definition, you don't want to protect them from everybody. It makes no sense to protect assets from their owners, or from other authorized individuals (including the trusted personnel who maintain the security system). In effect, then, all security systems need to allow people in, even as they keep people out. Designing a security system that accurately identifies, authenticates, and authorizes trusted individuals is highly complex and filled with nuance, but critical to security.
The Scalability Problem
Back in the mid-1990s, I worked for a company that developed multimedia kiosk demos. Our biggest client was Intel, and we often created demos that appeared in new PCs on the end-caps of major computer retailers such as CompUSA. At that time, performance was in demand for all application classes from business to consumer. We created demos that showed, for example, how much faster a spreadsheet would recalculate (you had to do that manually back then) on a new processor as compared with the previous year's processor. The differences were immediately noticeable to even a casual observer - and it mattered.
AI in Computer Games
If you've been following the game development scene, you've probably heard many remarks such as: "The main role of graphics in computer games will soon be over; artificial intelligence is the next big thing!" Although you should hardly buy into such statements, there is some truth in them. The quality of AI (artificial intelligence) is a high-ranking feature for game fans in making their purchase decisions and an area with incredible potential to increase players' immersion and fun.
Fun and Games:
Multi-Language Development
Computer games (or "electronic games" if you encompass those games played on console-class hardware) comprise one of the fastest-growing application markets in the world. Within the development community that creates these entertaining marvels, multi-language development is becoming more commonplace as games become more and more complex. Today, asking a development team to construct a database-enabled Web site with the requirement that it be written entirely in C++ would earn scornful looks and rolled eyes, but not long ago the idea that multiple languages were needed to accomplish a given task was scoffed at.
Massively Multiplayer Middleware
Wish is a multiplayer, online, fantasy role-playing game being developed by Mutable Realms. It differs from similar online games in that it allows tens of thousands of players to participate in a single game world (instead of the few hundred players supported by other games). Allowing such a large number of players requires distributing the processing load over a number of machines and raises the problem of choosing an appropriate distribution technology.
Game Development:
Harder Than You Think
The hardest part of making a game has always been the engineering. In times past, game engineering was mainly about low-level optimization - writing code that would run quickly on the target computer, leveraging clever little tricks whenever possible.
Black Box Debugging
It's all about what takes place at the boundary of an application.
Sink or Swim:
Know When It's Time to Bail
A diagnostic to help you measure organizational dysfunction and take action
Culture Surprises in Remote Software Development Teams
"When in Rome" doesn't help when your team crosses time zones, and your deadline doesn't.
Building Collaboration into IDEs
The Sun Never Sits on Distributed Development
People around the world can work around the clock on a distributed project, but the real challenge lies in taming the social dynamics.
Distributed Development:
Lessons Learned
Why repeat the mistakes of the past if you don't have to?
Uprooting Software Defects at the Source
Source code analysis is an emerging technology in the software industry that allows critical source code defects to be detected before a program runs.
Sentient Data Access via a Diverse Society of Devices
Today's ubiquitous computing environment cannot benefit from the traditional understanding of a hierarchical file system.
Nine IM Accounts and Counting
The key word with instant messaging today is interoperability. Various standards are in contention.
Broadcast Messaging:
Messaging to the Masses
This powerful form of communication has social implications as well as technical challenges.
Beyond Instant Messaging
Platforms and standards for these services must anticipate and accommodate future developments.
Reading, Writing, and Code
Forty years ago, when computer programming was an individual experience, the need for easily readable code wasn't on any priority list. Today, however, programming usually is a team-based activity, and writing code that others can easily decipher has become a necessity. Creating and developing readable code is not as easy as it sounds.
The Big Bang Theory of IDEs
Remember the halcyon days when development required only a text editor, a compiler, and some sort of debugger (in cases where the odd printf() or two alone didn't serve)? During the early days of computing, these were independent tools used iteratively in development's golden circle. Somewhere along the way we realized that a closer integration of these tools could expedite the development process. Thus was born the integrated development environment (IDE), a framework and user environment for software development that's actually a toolkit of instruments essential to software creation. At first, IDEs simply connected the big three (editor, compiler, and debugger), but nowadays most go well beyond those minimum requirements.
Modern System Power Management
The Advanced Configuration and Power Interface (ACPI) is the most widely used power and configuration interface for laptops, desktops, and server systems. It is also very complex, and its current specification weighs in at more than 500 pages. Needless to say, operating systems that choose to support ACPI require significant additional software support, up to and including fundamental OS architecture changes. The effort that ACPI's definition and implementation has entailed is worth the trouble because of how much flexibility it gives to the OS (and ultimately the user) to control power management policy and implementation.
Making a Case for Efficient Supercomputing
A supercomputer evokes images of "big iron" and speed; it is the Formula 1 racecar of computing. As we venture forth into the new millennium, however, I argue that efficiency, reliability, and availability will become the dominant issues by the end of this decade, not only for supercomputing, but also for computing in general.
Energy Management on Handheld Devices
Handheld devices are becoming ubiquitous and as their capabilities increase, they are starting to displace laptop computers - much as laptop computers have displaced desktop computers in many roles. Handheld devices are evolving from today's PDAs, organizers, cellular phones, and game machines into a variety of new forms. Although partially offset by improvements in low-power electronics, this increased functionality carries a corresponding increase in energy consumption. Second, as a consequence of displacing other pieces of equipment, handheld devices are seeing more use between battery charges. Finally, battery technology is not improving at the same pace as the energy requirements of handheld electronics.
The Inevitability of Reconfigurable Systems
The introduction of the microprocessor in 1971 marked the beginning of a 30-year stall in design methods for electronic systems. The industry is coming out of the stall by shifting from programmed to reconfigurable systems. In programmed systems, a linear sequence of configuration bits, organized into blocks called instructions, configures fixed hardware to mimic custom hardware. In reconfigurable systems, the physical connections among logic elements change with time to mimic custom hardware. The transition to reconfigurable systems will be wrenching, but this is inevitable as the design emphasis shifts from cost performance to cost performance per watt. Here's the story.
Getting Gigascale Chips:
Challenges and Opportunities in Continuing Moore's Law
Processor performance has increased by five orders of magnitude in the last three decades, made possible by following Moore's law - that is, continued technology scaling, improved transistor performance to increase frequency, additional (to avoid repetition) integration capacity to realize complex architectures, and reduced energy consumed per logic operation to keep power dissipation within limits. Advances in software technology, such as rich multimedia applications and runtime systems, exploited this performance explosion, delivering to end users higher productivity, seamless Internet connectivity, and even multimedia and entertainment.
Spam, Spam, Spam, Spam, Spam, the FTC, and Spam
A forum sponsored by the FTC highlights just how bad spam is, and how it's only going to get worse without some intervention.
Another Day, Another Bug
We asked our readers which tools they use to squash bugs. Here's what they said.
No Source Code? No Problem!
What if you have to port a program, but all you have is a binary?
Code Spelunking:
Exploring Cavernous Code Bases
Code diving through unfamiliar source bases is something we do far more often than write new code from scratch--make sure you have the right gear for the job.
Coding Smart: People vs. Tools
Tools can help developers be more productive, but they're no replacement for thinking.
Debugging in an Asynchronous World
Hard-to-track bugs can emerge when you can't guarantee sequential execution. The right tools and the right techniques can help.
Closed Source Fights Back
SCO vs. The World-What Were They Thinking?
Commercializing Open Source Software
Many have tried, a few are succeeding, but challenges abound.
The Age of Corporate Open Source Enlightenment
Like it or not, zealots and heretics are finding common ground in the open source holy war.
From Server Room to Living Room
How open source and TiVo became a perfect match
Storage Systems:
Not Just a Bunch of Disks Anymore
The sheer size and scope of data available today puts tremendous pressure on storage systems to perform in ways never imagined.
You Don't Know Jack about Disks
Whatever happened to cylinders and tracks?
Open Spectrum:
A Path to Ubiquitous Connectivity
Path to Ubiquitous Connectivity
Self-Healing Networks
Wireless networks that fix their own broken communication links may speed up their widespread acceptance.
Designing Portable Collaborative Networks
A middleware solution to keep pace with the ever-changing ways in which mobile workers collaborate.
The Family Dynamics of 802.11
The 802.11 family of standards is helping to move wireless LANs into promising new territory.
Caching XML Web Services for Mobility
In the face of unreliable connections and low bandwidth, caching may offer reliable wireless access to Web services.
The Future of WLAN
Overcoming the Top Ten Challenges in wireless networking--will it allow wide-area mesh networks to become ubiquitous?
Putting It All Together
Component integration is one of the tough challenges in embedded system design. Designers search for conservative design styles and reliable techniques for interfacing and verification.
Blurring Lines Between Hardware and Software
Software development for embedded systems clearly transcends traditional "programming" and requires intimate knowledge of hardware, as well as deep understanding of the underlying application that is to be implemented.
Division of Labor in Embedded Systems
You can choose among several strategies for partitioning an embedded application over incoherent processor cores. Here's practical advice on the advantages and pitfalls of each.
SoC: Software, Hardware, Nightmare, Bliss
System-on-a-chip design offers great promise by shrinking an entire computer to a single chip. But with the promise come challenges that need to be overcome before SoC reaches its full potential.
Programming Without a Net
Embedded systems programming presents special challenges to engineers unfamiliar with that environment.
Web Services: Promises and Compromises
Much of web services' initial promise will be realized via integration within the enterprise.
An Open Web Services Architecture
The name of the game is web services.
The Deliberate Revolution
Transforming Integration With XML Web Services