Moe, Larry, and Curly Braces

We've thus far discussed several of the primary pillars of Precision Modeling -- derivation expressions, conditional expressions, query expressions, and derived associations -- and we've used these to help out our hypothetical (and some might say homicidal) customer, arms4less.com ("where you can get regime change for small change!"). We've been specifying the business rule stating that customers' can pay a surcharge to prevent their enemies from purchasing any of the same weapons they're purchasing themselves. We did this by creating a derived association, enemyProducts, that contains the list of all products we are unable to sell to a particular customer, because they have already been purchased by enemies of the customer who have opted to pay the surcharge:

/enemyProducts : Product = enemies.orders.items[isExclusive].product

Now we just need to somehow use this association to prevent a customer from actually buying any of these enemyProducts. As with the many wonders of Precision Modeling we've previously discussed, we can specify such a requirement precisely and concisely (Hey, that rhymes. Our marketing people will be so pleased.) via a little Gorilla UML.

Notice the constraint expression we've attached to OrderItem:

{not product in order.customer.enemyProducts}

In UML, constraints are specified inside curly braces (many have postulated this proves once and for all that Grady Booch is a die-hard Three Stooges fan). Because this constraint is attached to OrderItem, it is specifying that any product associated with an OrderItem cannot also be included in the list of the order's customer's enemyProduct's list.

Our SME, Sarge, inquires as to how we plan to enforce this constraint? "If it gets violated, there needs to be -- you know -- consquences." We ensure him that we will be sure to thoroughly test our implementation of this constraint to ensure that the software indeed enforces it. Sarge sighs wistully as he removes his hand from a bulge beneath his flack jacket.

The enemy of my enemy is my customer

If you're new here, this post is part of a more-or-less continuous train-of-thought that begins here.

Today we continue looking at query expressions, which I introduced with much hullabaloo last time, but have thus far only given you the smallest taste. The glutinous among you are surely clamoring for more. So let's eat. But please stop using that use case diagram as a napkin. (Javadoc is much more absorbent).

Sarge has for the first time invited us to the headquarters of Arms4Less.com. After our blindfolds have been removed and our hands untied, we find ourselves seated around a large conference table in a windowless room. Charts around the wall show sales projections and troop strength estimates for the various markets Arms4Less is targeting over the next twelve months.

Last time we looked at how we can derive a value (isTrusted) using a query expression. Let's consider another such example. Sarge explains to us that Arms4Less prides itself on its high ethical standards. "Like if we sell a particular model of tank to one guy, for a small surcharge of 10%, we won't sell the same kinda tank to any of his enemies." We consider this requirement and realize that we need to prevent a customer from ordering any product that has been previously ordered by any of the customer's enemies. Now many of you are probably wonder how the heck we can express something like this on a lowly class diagram. Doesn't this require a more procedurally-oriented specification? Something with a little more get-up-and-go than a class diagram? Class diagrams just lay there, you say. We need something that moves, don't we?

Ha! I laugh at you (with all due respect of course). Do you think I would have wasted this much time blogging about this crap if we could go no further? Consider our latest version of the precision model, below:



We have added a new association, enemies, to keep track of a customer's enemies, and we have added a derived association called enemyProducts that references all the products purchased by a customer's enemies. Let me clarify a few things about "derived associations". First, you may be scratching your head about why we're calling enemyProducts an association at all, since it's not represented by a line on our class diagram. Notice however the type of the enemyProducts attribute. It's a Product. (In fact, as we'll discuss another time, it's a list of Products.)

There are multiple ways of representing associations in UML. For example, the following two representations are virtually equivalent.

On top, we see that the Employee has a BattleInjury reference called injuries. We see exactly the same thing with the Employee representation on the bottom, which contains a BattleInjury reference as well. The only difference is the notation being used. On the top we depict the reference with a composite association named injuries, whereas on the bottom, we decpict the reference with a nested attribute, also called injuries.

A derived association then is simply a derived attribute whose formula evaluates to a list of entities of some type. Let's look at the derivation expression for enemyProducts:

enemies.orders.items[isExclusive].product

In evaluating the value of this expression for a particular customer, we navigate through the customer's associated enemies, to their orders and then to the order's associated items. We then subset (query) the orders to select out those for which enemies have opted for an exclusive deal, and then we navigate from this resulting set to their associated products. The resulting list of products are the ones that we can't sell to the Customer.

The astute reader (or at least most who weren't out drinking last night) will have noticed we're essentially navigating multiple paths in parallel when we navigate enemies.orders.items since a customer can have many enemies, each in turn having many orders, and each of those orders having many items. The expression "unions" the results of navigating these parallel paths.

Now, you may recall earlier in this tirade we talked about the importance of precision. In fact, recall that all these techniques we're using comprise a methodology called Precision Modeling. You can rest assured that I'm not just whipping out arbitrary syntax for each of the formulas we've used. All of them are expressed with a formal expression language that are part of what we call Gorilla UML (GUML).

GUML not only has formal syntax definition. It executes! (Sarge says, "Cool! I execute too!") More on execution later. In our next little visit together, we'll actually look at using GUML to specify constraints. Using constraints we can complete our specification of Sarge's business rule preventing a product being sold to a customer's enemies.

The Axis of e-Ville

Having looked at how we can use derived and conditional expressions in our class diagrams, we are now ready to consider a construct so powerful that I shudder to think of what could happen if Iran, North Korea, or Microsoft should get hold of it. In fact, I'm taking a huge risk in revealing it here today. If this is my last post, it's a good bet that I'm in the hands of foreign agents trying to force me into assisting with a diabolical plot involving germs, radiation, and Tomcat 5.5.

Yes, today you are ready to learn about (cue some suitably momentous music) query expressions! Wait! Where are you going? Come back. I'm tellin' ya. Query expressions are some serious mojo!

Let us feel their power together by returning again to our consideration of the Arms4Less online superstore ("where you don't need to spend a lot to destroy a lot"). Sarge explains to us that they don't sell to just anybody. "For the really good stuff, you gotta be somebody we trust -- either somebody we know, or a friend of at least two people we know." We easily capture this notion of who's trusted as follows.



We add a Customer entity with an association, friends, referencing customers who are a customer's friends. We give Customer a boolean attribute, isSomebodyWeKnow, and as well as a derived attribute, isTrusted. The value of isTrusted is derived from the expression:

isSomebodyWeKnow or #friends[isSomebodyWeKnow] >= 2

The []-brackets enclose a selection expression. friends[isSomebodyWeKnow] selects all of the friends of some customer where isSomebodyWeKnow is true. The #-sign is a shorthand for "count". So, we can see that isTrusted will evaluate to true if the customer is somebody we know or if they have at least 2 friends we know.

When next we meet, we'll delve into the notion of query expressions more fully. In the meantime, beware of North Koreans toting pirated UML editors....

Conditional Love

Let's consider some other amusing things we can do with class diagrams. With a little tape, some string, and a pair of scissors, we can of course convert most class diagrams into a ceremonial tribal mask or a pirate hat. Another fun thing we can do is derive attribute values using conditional expressions.

Let's return to our example of the Arms4Less online superstore. We're having lunch with our SME, Sarge, and he's just finished telling us a funny story about a coup he helped stage a few years back. "So Johnny turns to me and says, whadaya mean, where's the king? Who do we got tied up in the back of your car?" We laugh appreciatively, and then turn the conversation back to the requirements for the store. Sarge explains how they want to offer various sales incentives, "like 20% off for quantities over 50". We can easily modify our model to reflect this requirement, by adding a new derived attribute called discount to OrderItem, and then modifying the derivation expression on total to reflect any discounting to be applied. We must also add the necessary non-derived attributes, discountQty and discount, to the Product, as shown.

Hopefully, you are beginning to see how much more you could be getting out of your class diagrams if you would just spend a little more time drawing pictures and a little less time writing code. I know how that can be tough with all the pressure you're getting from religious eXtremists, but as we shall soon see, Precision Requirements Modeling fits very nicely into an Agile approach that even the religious right can love.

In our next installment we'll really get this party started when we consider how to derive values from query expressions. I know the anticipation will be killing you between now and then, and I'd love to show you right now, but I gotta go write some code.

Let's not get physical

In my last post, I promised to reveal how class diagrams can be your secret weapon in the war against imprecise requirements. How can this be, you wonder? Sure, class diagrams capture entities, attributes, and associations, but they're completely incapable of expressing the hopes, fears, aspirations and unwarranted hostility of my end users. Surely we need all the subtlety and nuance of natural language to fully capture the essence of what we're being required to build? Surely we need to capture requirements using the flowing prose of use cases?

As we've discussed, precision is certainly something to be aspired to in requirements specifications, and yet, the legal profession provides compelling testimony to the horrible things that happen to beautiful prose when we impose precision upon it. Let's begin to consider some other things we can express clearly, unambiguously, and some might say beautifully, via everybody's favorite UML diagram.

Suppose we've been asked to build an online storefront offering stolen military hardware at low-low prices. The company, Arms4Less, was founded by several retired mercenaries now holed up somewhere in Honduras. Our primary contact is this crusty old guy we know only as "Sarge". The first thing he says the system needs to do is "store orders for stuff."

"You know", he says, "we sell stuff. Hand grenades, bazookas, napalm cannisters, that kinda thing. We'll need the usual shopping cart deal, letting customers tell us how much of each product they want." We sketch out the diagram below.
As you would expect, an Order consists of 1 or more (1..*) items, and each OrderItem specifies the quantity of some product being ordered. Each Product has a price. Now, hopefully you're regarding this model and saying, "Yeah? So? Why am I wasting time reading this stupid guy's blog when I could be learning Spring?" In a moment we'll make this model a lot more interesting. The diamond, by the way, signifies a "composite" relationship between an Order and its items, which essentially means that if we delete an Order, all of its items will get deleted too. (You would of course know what the diamond means if you'd been paying attention instead of doing email in that UML course your boss sent you to.) If the other symbols being used here are mysterious to you, go hang out with Ambler and come back when you're done.

A tremendously useful, but largely underused (Ambler's overview doesn't even mention them), facility of class diagrams is the specification of "derived" attributes. Derived attribute values are - uh - derived. That is, their values don't need to be stored since they can be completely determined from other data values. For example, Sarge explains that we need to total up each order and make the customer's got enough dough to cover it. "We do this for the customer's own protection", Sarge explains. "You don't wanna know what's gotta happen if the guy can't pay us." And indeed, we really don't wanna know until a later iteration. So, we add 2 derived attributes to the model to deal with the order total.



We've added a total attribute to both the Order and the OrderItem entities. Note that each has a leading slash, which means "derived" in UML. The OrderLine total is, not surprisingly, derived by multiplying the quantity by the price of the associated product (quantity*product.price). The Order total is the sum of the totals of its items (sum(items.total)). We mentioned earlier that derived attributes don't need to be stored, and indeed you can see how we can simply calculate these attributes' values on the fly. Certainly, however, there are situations in which we store derived attributes if, for example, they're too expensive to calculate dynamically. Such a decision however is purely an implementation choice. Logically, these attributes are derived.† This is all about what the system must do, not how it will do it. So let's keep our functional requirements logical. As Sarge might say, "You don't wanna get physical with me unless you're good and ready, computer boy."

Now, you're probably still a little underwhelmed by all this. But as we'll see in our next episode, we've only just scratched the surface of the expressive power of the friendly, little class diagram.

Real programmers don't eat diagrams

Yesterday I discussed how we developers routinely deal with imprecise requirements and while most of us find this to be annoying from time to time, overall we don't consider this a Big Problem. Or perhaps many of us do recognize this as a Big Problem, but have resigned ourselves to its seeming intractability. In any event, we can all agree that if it were possible for us to unambiguously "encode" requirements precisely, and could do this economically, it would surely be better than the current, preferred practice of getting a "pretty good idea" of what some business guy needs, code it up, and then sit down with the guy to see what we got wrong, and come away with a "pretty gooder idea" than we had previously. And then repeating this process over and over again until we get it right or until it's time to go live.

Most development projects begin with a fuzzy expression of requirements as articulated via some combination of text, pictures, and threatening emails by some end users, and end with a concrete expression of requirements as expressed in code and entreaties for forgiveness by programmers. As we discussed, there is currently no widely recognized middle ground between the fuzziness of warm, friendly natural language-based requirements specs, and cold, hard code. As a result, developers are left to interpret the fuzzy as their first step toward implementing a solution. Precision Requirements Modeling (PRM) is a set of techniques for transforming fuzzy requirements into a "code-ready" requirements specification. PRM allows requirements to be expressed with the precision of code, without getting bogged down in requirements details. Perhaps most tantalizingly, PRM specifications are themselves executable, which means we can let end users test drive a specification prior to developers becoming lost within the dense tangle of the implementation jungle, where they become increasingly isolated from civilization as they fashion primitive tools and frameworks, and engage in pitched warfare with competing tribes using .NET.

PRM begins with good-old-fashioned domain modeling. Virtually all O-O developers engage in some level of domain modeling whether they realize it or not. Every time you specify a class that represents a business concept, you're engaging in domain modeling. Even the staunchest of anti-modeling activists can often be caught in moments of weakness drawing UML-ish boxes and lines on a whiteboard. Anybody who has worked on anything but the most trivial of applications has come to recognize the usefulness of UML class diagrams or, at a minimum, entity relationship diagrams as a way of allowing members of a team to get a shared understanding of the major abstractions or data representations being used and how they relate to one another. Of course, you can always just fire up a debugger and trace through your buddy's code, but most of us would prefer a less - uh - extreme approach to integrating the disparate pieces of an implementation.

Class diagrams and ERDs are some of the best ways of communicating among a team an application's entities, their attributes, and their relationships to each other. In terms of attributes, most developers specify no more than names and types of what will be persisted. But as we shall soon see, class diagrams have the potential of communicating so much more if we would just let them. How well do you really know class diagrams? Sure you draw them occasionally. Sometimes you may even bring them to lunch with your colleagues. But if you would just get to really know them, you would see that they're among your best friends.

In our next installment, we'll get better acquainted with some class diagrams and see how, with some minor extensions, they can be used to unambiguously specify large portions of any requirements specification, and bridge the gap between warm, fuzzy business people and cold, hard programmers.

Just what we need -- more lawyers

One of the things you gotta love about "agile" is the name. Is it ever bad to be "agile"? Is there a good alternative (other than synonyms) for "agility"? Here at Gorilla Logic, we've devised what we modestly consider a breakthrough approach to requirements definition. "Precision Requirements Modeling" is the best our marketing department has been able to come up with as the moniker for this new approach. Although mind-numbingly accurate in its characterization, the name lacks a certain zing. Perhaps "Naked Precision Requirements Modeling" would be better.

While all developers want to be rich, thin, agile, and left alone, the same cannot be said of their wanting to be "precise". This is a curious thing when you consider that programming of course is nothing if not "precise". Code specifies precisely what the system will do. For the time being at least, computers don't have the ability to ponder a set of instructions and decide how best to carry them out. Programming code is literal, bereft of simile, metaphor, and clever double entendres. Functional requirements on the other hand are captured as natural language either because they're being written by non-programmers, or because they need to be reviewed and approved by non-programmers. Natural language tends to be pretty imprecise. (That's why we need programming languages.) Our industry has thus become accustomed to routinely working with imprecise requirements specifications.

Now some will argue that natural language can be quite precise. Aren't legal contracts shining examples of precise, natural language? Perhaps the world needs more lawyers. On the other hand, legal contracts tend to be fairly unintelligible to non-lawyers, and although I can't prove this, I will assert that most lawyers would make crummy programmers. Since non-lawyers are going to be writing the code, we probably need something other than legalese to communicate requirements.

Some will say, "aha"! This is precisely why striving for precision in requirements is a waste of time. "Those wacky business people can't possibly know if they want something or not until they see it running on their own desktops anyway". To a large extent this is of course true, but the number of cycles required to get things accepted by an end user will surely be reduced in direct proportion to how well we can understand the requirements prior to writing each round of code. Many theologians will say that although better requirements will reduce the cycles, they don't necessarily reduce the overall effort, since we're just substituting time spent requirements writing for time spent coding (and wouldn't you rather be writing code right now?). In fact, given what we've discussed about the imprecise nature of natural language, aren't we just better off capturing the requirements as code rather than trying to turn a requirements specification into a legal contract?

Perhaps. But this argument ignores the fact that when we code, we deal with far more than functional requirements. As much as 80% of code can be concerned with implementation details as opposed to business logic. (Actually, I just totally made up this statistic, but if you write code, I'm sure you'll agree that a very large proportion of code is solely concerned with plumbing). So the problem is that when we run off and code up a set of loosely defined requirements, we spend a whole lot of time dealing with implementation details that along with the business logic will need to be redone if our interpretation of those requirements is somehow incorrect.

Ah, but isn't this what "refactoring" is all about? The answer to that question is "no". Refactoring is about cleaning up an implementation. It's about avoiding over-engineering by just writing the damn code, and then restructuring that code when it becomes unwieldy, rather than building elaborate frameworks before we truly understand what our implementation requires. "Requirements refactoring" is a euphemism for getting things wrong. When we need to "refactor code" to reflect a requirements "clarification", we are simply rewriting code that was written to do something nobody actually wanted.

If there were only some way to "code" requirements without having to deal with implementation details. Then, we could indeed use code as our medium for capturing precise requirements, without wasting a lot of time on unneeded implementation should those requirements prove inaccurate. Hmmm, if only there were some way to capture requirements as code without implementation details.... Oh, that's right. There is a way. And coincidentally, it's what we've been working on for the last six years here at GL.

Searching for love in all the wrong use cases

Having established (ie, forcefully asserted) that it is generally Bad to engage in ritualistic development practices, but Good to adhere to a particular practice when it's well-suited to surmounting some challenge, let's consider the first snarling beast that leaps out to menace most young, impressionable development projects before they've even secured funding -- requirements definition.

Functional requirements are strange and mysterious things. Many programmers have never actually seen any, relying instead on rumor, innuendo or hearsay to guide their development objectives. For many, functional requirements are the only things keeping programming from being their dream vocation. Programming after all is, at its best, a beautifully artistic endeavor wherein we devise clever structures to manage the complexity of some problem, and transform chaos into order. Implementation "elegance" is regarded as one of the highest achievements we can attain as developers. As each new project begins, we think wistfully of the various forms of abstraction and all the intricate frameworks we have mastered over the years, and consider how, armed with the depth of our experience, this project will surely be different from our past ones. This time we'll build the thing Right, methodically assembling a great edifice from a series of simple building blocks. Each will be as cohesive as the nucleus of an atom, but as loosely coupled from one another as the grains of sand along the seashore. You kick off your shoes, and wander along the beach, smelling the salt air and thinking about that time under the boardwalk -- then, suddenly, a subject matter expert comes marching down the beach with several unruly business owners in tow, and all hell breaks loose.

Real-world requirements are seldom elegant. On the contrary, they tend to be maddeningly messy. For many applications, the hand-waving of some business stakeholder-guy can be sufficient for reaching an understanding of what needs to be built. But, alas, the majority of applications we build are far more "interesting" than these. Confronted by the terrifying unknown of some obscure workflow that has evolved over millions of years and countless mergers and acquisitions, many developers immediately fall back on their religious texts for guidance and comfort against the onrushing horror. They begin frantically writing user stories, drawing anatomically inaccurate stick figures, and in the dark of the night wondering why the gods of order and logic have forsaken them despite their best attempts to follow the good and righteous path documented and indexed in their most venerated religous text (second edition, totally updated with a new preface by the author).

Why must it be so? Why do business requirements, which on the surface seem so simple compared to the complexity of the software platforms we, as accomplished software developers, whip into submission almost every day, so often defy our best efforts to wrestle them to the ground though we pelt them with CRC cards and lock them in use cases? Is there no way to reduce the arbitrary tangle of business rules to some elegant expression of the problem that can return us to our idyllic afternoon at the seaside?

Grab a towel and some sunscreen and allow me to introduce you to Precision Requirements Modeling.

Too Much Religion

Programmers are a religious sort, unquestioningly engaging in ritualistic practices pronounced as "good" by the industry's high priests, and likewise shunning those practices branded as blasphemous. Everyone knows for example that mixing business logic with user interface code will damn your soul to Hell for all of eternity. Likewise, engaging in waterfall development leads to nowhere but poverty and pestilence. And each time you use dependency injection, an angel gets his wings.

The religious tenets of our industry of course have mostly arisen from hard-learned lessons about what seems to generally work and what seems generally not to. Still, it is clearly the case that there is no single path to project nirvana. Every development initiative grapples with a unique variety of technical, political, and sociological challenges. Some projects even succeed. It is of course natural that people examine the practices used by the really successful ones, and strive to emulate them to maximize their own chances for salvation.

Successful software development, alas, cannot be achieved by blindly following the prescriptions laid down in some sacred textbook or blog posting. The sacred texts do indeed provide valuable techniques discovered only after the spilling of vast amounts of blood by our ancestors who came down out of their mainframes to spread out across the network, converting green screens to rich clients, and procedures to objects. So, let us embrace the tools and techniques, but let us also recognize that the reason software projects fail is that there is no "Way" that one can simply follow to guarantee success.

Software development projects present us with a series of challenges. Each of those challenges can generally be addressed with some established set of tools and techniques. When we complete a successful project we are typically tempted to look back on the steps we took along the way and announce to our colleagues, "Ah, yes, truly we have found The Way. Let The Way be written down so that it be made repeatably so, and we can look forward to nothing but happy days for ourselves, our managers, and our manager's managers." And yet we then see others try to follow The Way, only to find misery, chaos, and even death. Okay, maybe not death.

So are we condemned then to roll the dice each time we undertake a new development initiative? I don't think so. But the solution is not religion. The solution is in learning a few highly useful techniques, but rather than applying them ritualistically, we should instead learn to assess the particular challenges presented at a particular juncture of a particular project, and to select among and adapt the various techniques to the challenge at hand.

The Way does not exist. This is The Only Way.