In a Perfect World....

In a perfect world, we could focus solely on implementing functional requirements and not have to do anything explicitly to deal with non-functional constraints such as keeping shared resources from becoming bottlenecks or contending with bandwidth limitations. It can be instructive to consider what an application would look like if there were no physical constraints. Imagine if we lived in a world with infinite memory, infinite processing capacity, unlimited bandwidth, zero down-time, and interesting television shows (Imagine really hard). In such a world, queries are blindingly fast , there is no limit to how often we interact with the back-end, and we can can constantly exchange enormous amounts of data if we need to. With such a platform, we are free to develop our application without any consideration of physical constraints. Each view can refresh the data it is displaying nearly instantaneously, and we can immediately transmit changes to the back-end, and immediately receive notifications of any errors encountered.

In such an environment, the need for a great many design constraints is removed, although not completely eliminated. Even with physical constraints removed, logical constraints remain. Still, removing the physical constraints certainly simplifies software construction by removing the need to manage limited resources efficiently. Of course, in the real world of enterprise applications, there is no such environment, but not every physical or logical constraint is an issue for every application. Low volume applications or those with very small and simple data requirements can often behave almost as if they were executing in the (near-)perfect world. Consider for example an application that allows a user to update their name, address, and credit card information. As soon as the user logs in, you can easily retrieve such a small volume of data immediately, and you wouldn't have to contend with the user's name and address being changed by someone else simulataneously. You can then allow the user to update any of the returned data, updating their name and address, deleting some credit cards and adding others. If there are few simultaneous users, we can transmit each change immediately to the server where we immediately run required edit checks against it and report any errors encountered back to the client for communication to the user via the UI. Such an approach allows us to write an application in what is arguably the most “natural” way, ie, the simplest way. Physical constraints, on the other hand, force us to commit "unnatural acts", ie, they force us to write code that is more complex than it needs to be from a purely logical perspective.

With RIA's, there are a few physical constraints which are fundamental at this point in time. The client application is a separate process (on a unique platform), and all communication with the server-side must be accomplished by messaging. It is this constraint, more than anything else, that distinguishes ERIA architecture from web applications, two-tier client server applications, or one-tier local applications. And speaking of one-tier applications, it is worth noting that virtually no enterprise application allows for the data to be persisted entirely on the client box. It is virtually always the case that we cannot rely on the client itself for safe storage of business records, and there are typically large amounts of non-client-specific data (such as a product catalog) that are usually too large to make client-side distribution practical.

Another constraint is the almost universal requirement that persistence be provided by a non-object-oriented datastore, typically a relational database. There is of course a fundamental impedance mismatch between object- and a relational-centric data structures that is at the heart of many unnatural programming acts, and although there have been many variations on “object databases” over the years, relational databases continue to dominate for good reasons we’ll explore in some depth another time. In a perfect world, queries would return graphs of objects rather than lists of rows. Fortunately, persistence management frameworks such as Hibernate automate a good deal of the acrobatics involved in object-relational mapping, but unfortunately are not (yet?) fully transparent to the application developer, requiring varying amounts of explicit coding.

Coding in the real world requires more effort than coding in the perfect world because we need to add code to work around physical constraints. It would seem to be self-evident that since it takes more effort to add more code, you should only do so when there is an actual requirement to do so. In other words, the default architecture for any ERIA is essentially the simple one described above. We retrieve all the data up-front. Operate on the data as necessary. And send edits to the back-end as they occur. In pondering each of the questions posed above, the default answer in each case is this straight-forward, perfect world architecture since it requires the least effort to build. In each case however it is also necessary to consider how physical constraints can force the undertaking of more complicated strategies, but again, it should be emphasized that in the absence of any such physical constraints, the simplicity of the perfect world approach is the way to go.

What is an architecture and why do you want one?

An architecture is essentially a set of design constraints imposed on an IT development project. You want one because unconstrained design leads to impedance mismatches among the various pieces of a system due to logical or physical incompatibilities among software components developed by different team members or third parties. The result is software that takes longer or costs more to build, or is more expensive to operate once it's deployed.

Impedance mismatches can manifest in a variety of ways.

• Model - Multiple, possibly inconsistent object definitions for the same logical entity
• API - Incompatible arguments or return results required or produced by libraries
• Performance - Inability of data sources to provide data as fast as required
• Locks - Contention resulting from different components having different expectations for how resources should be shared
• Transactions - Improperly serialized updates resulting in data being lost or overwritten with stale values
• Responsibility - Functionality needlessly duplicated or incorrectly assumed to be provided elsewhere
• Recovery - Components in the event of failure not coordinating properly to minimize downtime or data loss
• Operational - Software too difficult to modify in response to anticipated future needs

An architecture is sufficiently defined when you can turn a developer loose on developing some piece of functionality and if he or she obeys all the established design constraints, there is a minimal possibility of creating software that produces any of the above problems. If you're conceptually bothered by being constrained, preferring instead to program in wide open spaces with the wind blowing in your hair, perhaps you would prefer to instead think of architecture a form of freedom -- the freedom from choice. Once defined, an architecture frees developers from having to make (possibly incompatible) choices for how to implement an application's functionality, and instead devote most of their time to implementing functional requirements while minimizing the time spent hacking together glue to integrate with the rest of the team's code.

Here's a very high-level breakdown of a typical ERIA:

BackendService

In any non-trivial enterprise application development project, multiple developers will be working in tandem, and in an RIA project where the client and server platforms do not even use the same programming language, it is natural that developers will be divided into front-end and back-end teams. In order to reduce dependencies between client and server code development and allow the teams to work in relative independence on their respective components, it makes sense to explicitly define an API that will provide the linkage between the client and server code.

The BackendService is typically implemented using a webservice-type container, for example Tomcat and associated plugins, which provides a framework for authorizing users, managing session data, and communicating over web protocols such as REST, SOAP, or AMF.

UserSession

As with most traditional web applications, we require a container that maintains session state for each logged in user. Unless application users are anonymous, the session state will at a minimum contain user identity information required for access control. For many applications, code can be simplified by maintaining other session-specific information on the server, such as for example the current contents of the user's shopping cart.

UserIdentity

In most applications, we require user identity information in order to be able to complete requests received from a client. For example, when a user requests information for "my account", the server must need to somehow know which account to access. It is typically not an option to supply the account number for example as part of the request, since the client could be running a hacked application that allows the user to specify an arbitrary account number. When the user authenticates with the server, the server must associate the authorized user's identity with the session, and check each subsequent request against the that identity information in order to perform the requisite access control.

UserIdentity is typically implemented with role-based security frameworks such as JEE security implementations riding on LDAP or some other user directory store.

DomainObject

DomainObjects are instances of classes that directly model some aspect of our problem domain, such as Customers, Invoices, and Products. These objects are mostly created from input received from the client, or by being retrieved from a database or some external source, and additionally provide the non-UI-specific logic comprising our application.[Mention DDD?]

If we do our jobs right, most of the custom code we write on the backend will be domain-specific, having delegated most of the non-domain-specific plumbing to third-party frameworks.

PersistenceManager

The PersistenceManager caches previously retrieved data and collects updates triggered by the client until such time as a logical transaction is completed and can be committed to a database or external system. For relational databases, the PersistenceManager is typically implemented with a framework such as Hibernate. [Link to hibernate definition?]

BackEndServiceAPI

Clients access the BackEndService via a library that provides a client-code-friendly interface. This BackEndServiceAPI is often no more than a wrapper that exposes the BackEndService interface in the client's native language, and does little more than marshal arguments as necessary to call whatever remoting framework is being used for backend communication, and set up whatever callbacks are needed in cases where results are returned asynchronously.

The BackEndServiceAPI would usually be defined as an interface for which a mock implementation can be used as a stand-in so that the front-end team has the option of proceeding with development ahead of the back-end implementation being complete.

DomainDTO

In a typical RIA, we cannot actually return full-blown objects from the server to the client or vice versa. Instead, we can only transmit an object's primitive property values. For example, if the client requests a particular Customer, the server does not return an actual instance of a DomainObject, but instead returns a set of values such as name, address, and account number. While its possible to construct an actual DomainObject instance from these returned values, it is often unnecessary since an RIA in many cases is merely providing for the display and editing of such primitive values, delegating any requisite business logic back to the server-based DomainObjects. DomainDTO's are objects that provide holders for these values, but don't necessarily provide any "behavior" in the form of methods on the objects. It is of course also common to selectively replicate some DomainObject logic on the client, especially simple field edits, and in some cases we might even replicate complex logic on the client, so we're using the term DTO loosely here. [Link to DTO definition?]

[Diagram showing a DomainObject vs corresponding DomainDTO?]

CacheManager

With a thin client, client-side state does not usually consist of anymore than the contents of some html form or table. Each new view usually requires a round-trip to the server to retrieve both the definition of the view (ie, an html page), and the data presented within it. Even just changing the sort sequence of the rows in a table require re-retrieving a fully formatted table of data from a server. One of the big advantages in an RIA is that we can provide a much more interactive and responsive user experience by not requiring the constant back-and-forth of data and markup with the server. Instead we can retrieve (possibly large amounts of) data just once, display it in a variety of ways, allow the user to edit it, and send updates to the server only after a transaction is logically complete.

The CacheManager keeps track of what we've already retrieved from the server so that we don't incur unnecessary overhead of re-retrieving the same data, and tracks what objects are changed ("dirty") and require transmission back to the server. The sophistication of CacheManagers can vary greatly from one application to another, consisting of little more than hash tables of objects by type [Include a simple diagram of this?], or as much as client-side relational databases.

ViewComponent

The view components are the widgets that provide for the display and editing data, usually DomainDTO's retrieved from the local cache. By ViewComponents, we don't mean low-level UI components like buttons and textfields, but semantically rich composite components such as for example a Customer Creation Form or an Order History Table.

EventBus

RIA platforms are event-driven user interface frameworks where ViewComponents emit and respond to asynchronous notifications such as "User Wants to Quit" or "User Has Updated His Credit Card Info". While such event-driven models are ideally suited to coordinating the asynchronous update and display of data across possibly multiple views, event-driven programs can degenerate into a tangle of interdependencies among components dispatching and listening for each other's events. An EventBus organizes event handling around a hub-and-spoke where events are logically broadcast throughout the application. In this way, brittle point-to-point connections are replaced by a robust, centralized switchboard for application events.

Application

The Application is "everything else". It's the code necessary to integrate the BackEndAPI, the ViewComponents, and the CacheManager into the functioning whole experienced by the user. The Application for example binds particular views to particular DTO's, and invokes particular BackEndAPI methods in response to user interactions.

Can I just go build my application now?

Having dilligently studied the above description, you may be chomping at the bit to put the above principles to work and begin building your next RIA. Unfortunately, the above is not the answer. It only frames the questions more clearly. Chiefly:

• Whether to locate logic on the client or server?
• How much data to cache locally on the client?
• When should data be transmitted from the client to the server?
• When should data be persisted to a database?
• How should view components be made aware of updates to what is being displayed?
• What is the right level of granularity for remote operations?

What's that you say? You came here for answers? Fear not. All will be revealed. But not this afternoon....

The Year of Coding Dangerously

You've got an appointment set up with your customer at a lunch place down by the water. You get there early, sit down at a booth and check the place out. There's a beautiful woman sitting at a table near the door, watching a tiny movie on her iPhone. You think, where have I seen her before? Is she from a rival consulting outfit? Could it be you're being followed? No, no, you think. Just paranoia. On the road too long.

Your customer arrives and sits down heavily on the other side of the table. He pushes a sky blue folder toward you and orders a patty melt. Inside you find a PowerPoint deck and flip through it, amazed that people still use such cheesy clipart. After a minute you look up, and trying to control your voice manage to say, "We can't do it in time." You reach for a cigarette but then realize you don't smoke. So instead you pick up a french fry.

Your customer gets an impatient look on his face and says, "Look, negotiations broke down in Helsinki. It was bad. They quit the consortium. They're not going with the standard. We've gotta do it their way. How bad could their proprietary stuff really be?".

Just then, an explosion rips through the building. You find yourself lying face up in the parking lot just as a helicopter lands and two guys wearing flak jackets with your company's logo jump out and pull you inside. As you fly off, you see the entire island being consumed in a volcanic eruption and sinking slowly into the sea. Just before you pass out again you think, next time I gotta get a local gig.

I'm sure we've all had experiences like this one. As application developers, we live in a world of excitement, intrigue and suspense. Yes, danger lurks around every turn. Sometimes it arrives with a whisper. A never-before-seen error message appears on a console, or a process turns up dead. Other times it comes with a bang. A load balancer goes beserk and the Big Board in the call center lights up like a pinball machine. The lives of thousands of active sessions hang by a thread. Your cell phone rings. The president needs an update....

As asserted in our last post, architecture can be seen as a a set of implementation constraints selected to mitigate risk. There are other ways to think about architecture, but this definition works well in that it emphasizes the "why" rather than the "what" of architecture. By focusing on the "why", we can better determine the "how much" and define no more architecture than necessary to deal with the risk profile of a particular project.

Architecture alone is of course insufficient for mitigating every kind of risk we might encounter on a project. We also need things like legal contracts, customer expectation management, and adequate QA testing, to name a few. Most of those other things are not, strictly speaking, technical risks, and so are not typically the province of application developers, being handled instead by people like lawyers, project managers, and that really annoying guy in the purchasing department.

In defining an Enterprise RIA Architecture, we will begin by identifying the most common risks faced by most development efforts and define our architecture in the context of providing mitigation strategies for those risks.

To be clear, what we mean by "Enterprise RIA" (ERIA) is any application in which a rich client user interface accesses a set of back-end services to execute transactions. ERIA's are characterized much more by this back-end interaction requirement than by any particular attributes of the user interface itself. Because the user interface of an ERIA can vary so dramatically from application to appllication, there's not much we'll say about the architecture of the presentation portion of an application, which might consist of anything from simple forms to sophisticated 3D aimations. We are much more concerned with how the presentation portion handles information and transactions in concert with back-end systems, where the back-end systems are information- or transaction-centric.

What, you may wonder, makes back-end interaction in an RIA fundamentally different from good-old-fashioned J2EE architecture? The answer is that in a typical J2EE application, virtually all the logic lives in the back-end. Yes, we can use JavaScript to execute logic on the client platform directly, but since, by definition, non-RIA applications provide non-rich UI functionality, that logic is typically limited and simple.

In an RIA, however, we are building full-blown client-side applications with client-side logic that can be quite complex. RIA clients tend to be highly stateful, oftentimes maintaining relatively large amounts of data client-side that create unique problems around synchronization with back-end systems of record. As we shall see, such front-end to back-end state synchronization is one of the major technical requirements that drive many of our architectural decisions.

In our next installment, we'll look more closely at the risks confronting most ERIA projects. Until then, watch your back-end and for godsakes make sure you're not being followed by agents of rival consulting outfits....

Architecture, smarchitecture.

Now begins our discussion of Rich Internet Application Architecture. This is a work in progress, which is to say that what is below is really a draft of a post, but we're exposing the salami-making process to invite collaboration from our friends and lovers.

The first thing we need to discuss is what we mean by "architecture", since this is surely an overloaded term in our industry. It seems to mostly say something about how a system's functionality (domain-specific and otherwise) is partitioned both logically and physically across application code, software platforms, and physical hosts. For the purposes of this discussion we'll define architecture as any set of implementation constraints imposed on a team during some project.

By this definition, a project without a pre-defined architecture is one in which their might be a "design", for example consisting of major classes and how they interact with each other, and choices about what platforms things will run on, but beyond that it is left up to each developer to determine how to implement each piece of functionality assigned to their work queue. We say that the project has no pre-defined architecture because one will almost inevitably "emerge" as various mechanisms and partitioning strategies get worked out and discussed among the team, and people naturally clone already working code when they need to do something similar. It could be argued that the resulting "architecture" is one that is tightly adapted to meet the actual (emerging) needs for one, rather than something possibly over-engineered beyond what we find is really needed after we've been coding on the project for a while.

Of course the real choice is not a binary one between having a pre-defined architecture and not having one. We can rigorously define some aspects of a system's architecture while letting other aspects emerge. But that still leaves the question: "How much architecture do we need?"

I propose that the answer to that question should be "Just enough to mitigate risk" and in summary will close with the proposition that:

A system's architecture definition should consist of the minimal set of implementation constraints needed to mitigate implementation risks.

In our next installment, we'll discuss the most commons risks confronting Enterprise RIA system implementations.

A sketch of a Rich Internet Application Architecture

Just another code-slinging CEO

I am a fossil, or more accurately, I should say that I'm a fossil record. Or perhaps the better metaphor would be that I'm like one of those ice cores geologists drill from the depths of the Arctic. If you were to sink a drill deep into my head (and I know many of my former colleagues would like to), you would find evidence of the many fads, trends, and revolutions that have constantly reshaped corporate IT over the past 30 years recorded in my brain like the stratified deposits within an ice core. By examining such ice cores, geologists arrive at a deeper understanding of our planet, and by examining the ice core in my brain, I will endeavor for us to arrive at a deeper understanding of Planet IT. I have not only been present for the many tectonic shifts that periodically rock our industry, I have usually been standing directly on the fault line, intimately involved as a leader, manager, and practitioner of application development.

I want to emphasize the "practioner" part of what I just said. Years ago, my colleagues and I would tell each other (half-) jokingly, "Don't trust anybody who doesn't log on". This was shorthand for our belief (which I still hold) that anybody who no longer understands the technology is inherently ineffective in directing IT initiatives. This does not mean that management needs to write code, but management does need to understand a fairly large body of key technical principles since these significantly impact the planning and execution of any IT project.

I myself have continued to write code in spite of having spent many years in fairly senior management positions (at Sun, for example, I managed 300 people). Hopefully as I delve into the subtleties of Rich Internet Application Architecture in the following series of posts, you will trust that my views are derived not just from a consideration of battles viewed from the safety of an underground bunker back at central command, but also while engaged in fierce trench warfare myself.

This experience, coupled with the ice core in my brain that informs my views with a deep (and painful) appreciation of all that has gone (wrong) before hopefully convinces you that my insights are more valuable than what you would find in a random blog post. (And yes, I realize that this is itself a random blog post and my last comment was a referential recursion of sorts that probably blew the stack of several unsuspecting readers).

In any event, I do hope you'll join me in my next few posts for an exploration of Rich Internet Application Architecture (I hesitate to call this RIAA since it's the RIAA that sues people for downloading Britney Spears singles), but before we begin, let's all go write some code.

The Tower of Babbage

In this post about an Eric Evans presentation, Jon Rose mentions how Eric needed to clarify that the intention is not for Ubiquitous Languages to be enterprise-wide. UL's are established across project teams, not organizations. I was initially surprised that such a clarification would be necessary since this seems obvious, but then I remembered life in the late eighties....

Love Shack was a smash hit and Enterprise Data Models were all the rage.

Much like the Arthur Clarke science fiction story in which a monastery of monks fulfills the purpose of the universe by recording with a computer every known name of God (when they were finished "overhead, without any fuss, the stars were going out"), the idea was that if we could just catalog EVERY data entity, attribute, and association across the ENTIRE enterprise, then surely we would come to know our domain, our applications, our users, indeed our very inner souls better, and thus build better software faster since all of the various warring factions in IT would finally speak the one, true tongue.

At my company, PaineWebber, several monkish DBA's undertook this task for several years, compiling an ever growing glossary of ever greater weight and size. One night in a dream (or perhaps it was after a series of lunches with a leggy sales rep) our CIO realized he could accelerate our rendevous with destiny by buying somebody else's financial services datamodel. And so it was that for the low, low price of $1.5 million, we purchased a great many binders from First Boston containing a great multitude of boxes and lines.

From time to time, developers would come to seek the truth of the binders. Like astrologers poring over charts of the stars, they would look for signs in the many boxes and lines, looking for clues to unlock the secrets of their particular problem domain. But, alas, while many of the boxes and lines bore striking resemblances to actual people, places, and things from the known world, there would also be striking differences from the reality they knew, and in any case, the detail of the models was simply too overwhelming -- or perhaps it was just too magnificent.

And so we went on as before, speaking our own local dialects and doing our best to communicate with neighboring tribes, using extract files like smoke signals, reeking with the smell of EBCDIC.

At least the stars didn't wink out.

Moe, Larry, and Curly Braces

We've thus far discussed several of the primary pillars of Precision Modeling -- derivation expressions, conditional expressions, query expressions, and derived associations -- and we've used these to help out our hypothetical (and some might say homicidal) customer, arms4less.com ("where you can get regime change for small change!"). We've been specifying the business rule stating that customers' can pay a surcharge to prevent their enemies from purchasing any of the same weapons they're purchasing themselves. We did this by creating a derived association, enemyProducts, that contains the list of all products we are unable to sell to a particular customer, because they have already been purchased by enemies of the customer who have opted to pay the surcharge:

/enemyProducts : Product = enemies.orders.items[isExclusive].product

Now we just need to somehow use this association to prevent a customer from actually buying any of these enemyProducts. As with the many wonders of Precision Modeling we've previously discussed, we can specify such a requirement precisely and concisely (Hey, that rhymes. Our marketing people will be so pleased.) via a little Gorilla UML.

Notice the constraint expression we've attached to OrderItem:

{not product in order.customer.enemyProducts}

In UML, constraints are specified inside curly braces (many have postulated this proves once and for all that Grady Booch is a die-hard Three Stooges fan). Because this constraint is attached to OrderItem, it is specifying that any product associated with an OrderItem cannot also be included in the list of the order's customer's enemyProduct's list.

Our SME, Sarge, inquires as to how we plan to enforce this constraint? "If it gets violated, there needs to be -- you know -- consquences." We ensure him that we will be sure to thoroughly test our implementation of this constraint to ensure that the software indeed enforces it. Sarge sighs wistully as he removes his hand from a bulge beneath his flack jacket.

The enemy of my enemy is my customer

If you're new here, this post is part of a more-or-less continuous train-of-thought that begins here.

Today we continue looking at query expressions, which I introduced with much hullabaloo last time, but have thus far only given you the smallest taste. The glutinous among you are surely clamoring for more. So let's eat. But please stop using that use case diagram as a napkin. (Javadoc is much more absorbent).

Sarge has for the first time invited us to the headquarters of Arms4Less.com. After our blindfolds have been removed and our hands untied, we find ourselves seated around a large conference table in a windowless room. Charts around the wall show sales projections and troop strength estimates for the various markets Arms4Less is targeting over the next twelve months.

Last time we looked at how we can derive a value (isTrusted) using a query expression. Let's consider another such example. Sarge explains to us that Arms4Less prides itself on its high ethical standards. "Like if we sell a particular model of tank to one guy, for a small surcharge of 10%, we won't sell the same kinda tank to any of his enemies." We consider this requirement and realize that we need to prevent a customer from ordering any product that has been previously ordered by any of the customer's enemies. Now many of you are probably wonder how the heck we can express something like this on a lowly class diagram. Doesn't this require a more procedurally-oriented specification? Something with a little more get-up-and-go than a class diagram? Class diagrams just lay there, you say. We need something that moves, don't we?

Ha! I laugh at you (with all due respect of course). Do you think I would have wasted this much time blogging about this crap if we could go no further? Consider our latest version of the precision model, below:



We have added a new association, enemies, to keep track of a customer's enemies, and we have added a derived association called enemyProducts that references all the products purchased by a customer's enemies. Let me clarify a few things about "derived associations". First, you may be scratching your head about why we're calling enemyProducts an association at all, since it's not represented by a line on our class diagram. Notice however the type of the enemyProducts attribute. It's a Product. (In fact, as we'll discuss another time, it's a list of Products.)

There are multiple ways of representing associations in UML. For example, the following two representations are virtually equivalent.

On top, we see that the Employee has a BattleInjury reference called injuries. We see exactly the same thing with the Employee representation on the bottom, which contains a BattleInjury reference as well. The only difference is the notation being used. On the top we depict the reference with a composite association named injuries, whereas on the bottom, we decpict the reference with a nested attribute, also called injuries.

A derived association then is simply a derived attribute whose formula evaluates to a list of entities of some type. Let's look at the derivation expression for enemyProducts:

enemies.orders.items[isExclusive].product

In evaluating the value of this expression for a particular customer, we navigate through the customer's associated enemies, to their orders and then to the order's associated items. We then subset (query) the orders to select out those for which enemies have opted for an exclusive deal, and then we navigate from this resulting set to their associated products. The resulting list of products are the ones that we can't sell to the Customer.

The astute reader (or at least most who weren't out drinking last night) will have noticed we're essentially navigating multiple paths in parallel when we navigate enemies.orders.items since a customer can have many enemies, each in turn having many orders, and each of those orders having many items. The expression "unions" the results of navigating these parallel paths.

Now, you may recall earlier in this tirade we talked about the importance of precision. In fact, recall that all these techniques we're using comprise a methodology called Precision Modeling. You can rest assured that I'm not just whipping out arbitrary syntax for each of the formulas we've used. All of them are expressed with a formal expression language that are part of what we call Gorilla UML (GUML).

GUML not only has formal syntax definition. It executes! (Sarge says, "Cool! I execute too!") More on execution later. In our next little visit together, we'll actually look at using GUML to specify constraints. Using constraints we can complete our specification of Sarge's business rule preventing a product being sold to a customer's enemies.

The Axis of e-Ville

Having looked at how we can use derived and conditional expressions in our class diagrams, we are now ready to consider a construct so powerful that I shudder to think of what could happen if Iran, North Korea, or Microsoft should get hold of it. In fact, I'm taking a huge risk in revealing it here today. If this is my last post, it's a good bet that I'm in the hands of foreign agents trying to force me into assisting with a diabolical plot involving germs, radiation, and Tomcat 5.5.

Yes, today you are ready to learn about (cue some suitably momentous music) query expressions! Wait! Where are you going? Come back. I'm tellin' ya. Query expressions are some serious mojo!

Let us feel their power together by returning again to our consideration of the Arms4Less online superstore ("where you don't need to spend a lot to destroy a lot"). Sarge explains to us that they don't sell to just anybody. "For the really good stuff, you gotta be somebody we trust -- either somebody we know, or a friend of at least two people we know." We easily capture this notion of who's trusted as follows.



We add a Customer entity with an association, friends, referencing customers who are a customer's friends. We give Customer a boolean attribute, isSomebodyWeKnow, and as well as a derived attribute, isTrusted. The value of isTrusted is derived from the expression:

isSomebodyWeKnow or #friends[isSomebodyWeKnow] >= 2

The []-brackets enclose a selection expression. friends[isSomebodyWeKnow] selects all of the friends of some customer where isSomebodyWeKnow is true. The #-sign is a shorthand for "count". So, we can see that isTrusted will evaluate to true if the customer is somebody we know or if they have at least 2 friends we know.

When next we meet, we'll delve into the notion of query expressions more fully. In the meantime, beware of North Koreans toting pirated UML editors....

Conditional Love

Let's consider some other amusing things we can do with class diagrams. With a little tape, some string, and a pair of scissors, we can of course convert most class diagrams into a ceremonial tribal mask or a pirate hat. Another fun thing we can do is derive attribute values using conditional expressions.

Let's return to our example of the Arms4Less online superstore. We're having lunch with our SME, Sarge, and he's just finished telling us a funny story about a coup he helped stage a few years back. "So Johnny turns to me and says, whadaya mean, where's the king? Who do we got tied up in the back of your car?" We laugh appreciatively, and then turn the conversation back to the requirements for the store. Sarge explains how they want to offer various sales incentives, "like 20% off for quantities over 50". We can easily modify our model to reflect this requirement, by adding a new derived attribute called discount to OrderItem, and then modifying the derivation expression on total to reflect any discounting to be applied. We must also add the necessary non-derived attributes, discountQty and discount, to the Product, as shown.

Hopefully, you are beginning to see how much more you could be getting out of your class diagrams if you would just spend a little more time drawing pictures and a little less time writing code. I know how that can be tough with all the pressure you're getting from religious eXtremists, but as we shall soon see, Precision Requirements Modeling fits very nicely into an Agile approach that even the religious right can love.

In our next installment we'll really get this party started when we consider how to derive values from query expressions. I know the anticipation will be killing you between now and then, and I'd love to show you right now, but I gotta go write some code.

Let's not get physical

In my last post, I promised to reveal how class diagrams can be your secret weapon in the war against imprecise requirements. How can this be, you wonder? Sure, class diagrams capture entities, attributes, and associations, but they're completely incapable of expressing the hopes, fears, aspirations and unwarranted hostility of my end users. Surely we need all the subtlety and nuance of natural language to fully capture the essence of what we're being required to build? Surely we need to capture requirements using the flowing prose of use cases?

As we've discussed, precision is certainly something to be aspired to in requirements specifications, and yet, the legal profession provides compelling testimony to the horrible things that happen to beautiful prose when we impose precision upon it. Let's begin to consider some other things we can express clearly, unambiguously, and some might say beautifully, via everybody's favorite UML diagram.

Suppose we've been asked to build an online storefront offering stolen military hardware at low-low prices. The company, Arms4Less, was founded by several retired mercenaries now holed up somewhere in Honduras. Our primary contact is this crusty old guy we know only as "Sarge". The first thing he says the system needs to do is "store orders for stuff."

"You know", he says, "we sell stuff. Hand grenades, bazookas, napalm cannisters, that kinda thing. We'll need the usual shopping cart deal, letting customers tell us how much of each product they want." We sketch out the diagram below.
As you would expect, an Order consists of 1 or more (1..*) items, and each OrderItem specifies the quantity of some product being ordered. Each Product has a price. Now, hopefully you're regarding this model and saying, "Yeah? So? Why am I wasting time reading this stupid guy's blog when I could be learning Spring?" In a moment we'll make this model a lot more interesting. The diamond, by the way, signifies a "composite" relationship between an Order and its items, which essentially means that if we delete an Order, all of its items will get deleted too. (You would of course know what the diamond means if you'd been paying attention instead of doing email in that UML course your boss sent you to.) If the other symbols being used here are mysterious to you, go hang out with Ambler and come back when you're done.

A tremendously useful, but largely underused (Ambler's overview doesn't even mention them), facility of class diagrams is the specification of "derived" attributes. Derived attribute values are - uh - derived. That is, their values don't need to be stored since they can be completely determined from other data values. For example, Sarge explains that we need to total up each order and make the customer's got enough dough to cover it. "We do this for the customer's own protection", Sarge explains. "You don't wanna know what's gotta happen if the guy can't pay us." And indeed, we really don't wanna know until a later iteration. So, we add 2 derived attributes to the model to deal with the order total.



We've added a total attribute to both the Order and the OrderItem entities. Note that each has a leading slash, which means "derived" in UML. The OrderLine total is, not surprisingly, derived by multiplying the quantity by the price of the associated product (quantity*product.price). The Order total is the sum of the totals of its items (sum(items.total)). We mentioned earlier that derived attributes don't need to be stored, and indeed you can see how we can simply calculate these attributes' values on the fly. Certainly, however, there are situations in which we store derived attributes if, for example, they're too expensive to calculate dynamically. Such a decision however is purely an implementation choice. Logically, these attributes are derived.† This is all about what the system must do, not how it will do it. So let's keep our functional requirements logical. As Sarge might say, "You don't wanna get physical with me unless you're good and ready, computer boy."

Now, you're probably still a little underwhelmed by all this. But as we'll see in our next episode, we've only just scratched the surface of the expressive power of the friendly, little class diagram.

Real programmers don't eat diagrams

Yesterday I discussed how we developers routinely deal with imprecise requirements and while most of us find this to be annoying from time to time, overall we don't consider this a Big Problem. Or perhaps many of us do recognize this as a Big Problem, but have resigned ourselves to its seeming intractability. In any event, we can all agree that if it were possible for us to unambiguously "encode" requirements precisely, and could do this economically, it would surely be better than the current, preferred practice of getting a "pretty good idea" of what some business guy needs, code it up, and then sit down with the guy to see what we got wrong, and come away with a "pretty gooder idea" than we had previously. And then repeating this process over and over again until we get it right or until it's time to go live.

Most development projects begin with a fuzzy expression of requirements as articulated via some combination of text, pictures, and threatening emails by some end users, and end with a concrete expression of requirements as expressed in code and entreaties for forgiveness by programmers. As we discussed, there is currently no widely recognized middle ground between the fuzziness of warm, friendly natural language-based requirements specs, and cold, hard code. As a result, developers are left to interpret the fuzzy as their first step toward implementing a solution. Precision Requirements Modeling (PRM) is a set of techniques for transforming fuzzy requirements into a "code-ready" requirements specification. PRM allows requirements to be expressed with the precision of code, without getting bogged down in requirements details. Perhaps most tantalizingly, PRM specifications are themselves executable, which means we can let end users test drive a specification prior to developers becoming lost within the dense tangle of the implementation jungle, where they become increasingly isolated from civilization as they fashion primitive tools and frameworks, and engage in pitched warfare with competing tribes using .NET.

PRM begins with good-old-fashioned domain modeling. Virtually all O-O developers engage in some level of domain modeling whether they realize it or not. Every time you specify a class that represents a business concept, you're engaging in domain modeling. Even the staunchest of anti-modeling activists can often be caught in moments of weakness drawing UML-ish boxes and lines on a whiteboard. Anybody who has worked on anything but the most trivial of applications has come to recognize the usefulness of UML class diagrams or, at a minimum, entity relationship diagrams as a way of allowing members of a team to get a shared understanding of the major abstractions or data representations being used and how they relate to one another. Of course, you can always just fire up a debugger and trace through your buddy's code, but most of us would prefer a less - uh - extreme approach to integrating the disparate pieces of an implementation.

Class diagrams and ERDs are some of the best ways of communicating among a team an application's entities, their attributes, and their relationships to each other. In terms of attributes, most developers specify no more than names and types of what will be persisted. But as we shall soon see, class diagrams have the potential of communicating so much more if we would just let them. How well do you really know class diagrams? Sure you draw them occasionally. Sometimes you may even bring them to lunch with your colleagues. But if you would just get to really know them, you would see that they're among your best friends.

In our next installment, we'll get better acquainted with some class diagrams and see how, with some minor extensions, they can be used to unambiguously specify large portions of any requirements specification, and bridge the gap between warm, fuzzy business people and cold, hard programmers.

Just what we need -- more lawyers

One of the things you gotta love about "agile" is the name. Is it ever bad to be "agile"? Is there a good alternative (other than synonyms) for "agility"? Here at Gorilla Logic, we've devised what we modestly consider a breakthrough approach to requirements definition. "Precision Requirements Modeling" is the best our marketing department has been able to come up with as the moniker for this new approach. Although mind-numbingly accurate in its characterization, the name lacks a certain zing. Perhaps "Naked Precision Requirements Modeling" would be better.

While all developers want to be rich, thin, agile, and left alone, the same cannot be said of their wanting to be "precise". This is a curious thing when you consider that programming of course is nothing if not "precise". Code specifies precisely what the system will do. For the time being at least, computers don't have the ability to ponder a set of instructions and decide how best to carry them out. Programming code is literal, bereft of simile, metaphor, and clever double entendres. Functional requirements on the other hand are captured as natural language either because they're being written by non-programmers, or because they need to be reviewed and approved by non-programmers. Natural language tends to be pretty imprecise. (That's why we need programming languages.) Our industry has thus become accustomed to routinely working with imprecise requirements specifications.

Now some will argue that natural language can be quite precise. Aren't legal contracts shining examples of precise, natural language? Perhaps the world needs more lawyers. On the other hand, legal contracts tend to be fairly unintelligible to non-lawyers, and although I can't prove this, I will assert that most lawyers would make crummy programmers. Since non-lawyers are going to be writing the code, we probably need something other than legalese to communicate requirements.

Some will say, "aha"! This is precisely why striving for precision in requirements is a waste of time. "Those wacky business people can't possibly know if they want something or not until they see it running on their own desktops anyway". To a large extent this is of course true, but the number of cycles required to get things accepted by an end user will surely be reduced in direct proportion to how well we can understand the requirements prior to writing each round of code. Many theologians will say that although better requirements will reduce the cycles, they don't necessarily reduce the overall effort, since we're just substituting time spent requirements writing for time spent coding (and wouldn't you rather be writing code right now?). In fact, given what we've discussed about the imprecise nature of natural language, aren't we just better off capturing the requirements as code rather than trying to turn a requirements specification into a legal contract?

Perhaps. But this argument ignores the fact that when we code, we deal with far more than functional requirements. As much as 80% of code can be concerned with implementation details as opposed to business logic. (Actually, I just totally made up this statistic, but if you write code, I'm sure you'll agree that a very large proportion of code is solely concerned with plumbing). So the problem is that when we run off and code up a set of loosely defined requirements, we spend a whole lot of time dealing with implementation details that along with the business logic will need to be redone if our interpretation of those requirements is somehow incorrect.

Ah, but isn't this what "refactoring" is all about? The answer to that question is "no". Refactoring is about cleaning up an implementation. It's about avoiding over-engineering by just writing the damn code, and then restructuring that code when it becomes unwieldy, rather than building elaborate frameworks before we truly understand what our implementation requires. "Requirements refactoring" is a euphemism for getting things wrong. When we need to "refactor code" to reflect a requirements "clarification", we are simply rewriting code that was written to do something nobody actually wanted.

If there were only some way to "code" requirements without having to deal with implementation details. Then, we could indeed use code as our medium for capturing precise requirements, without wasting a lot of time on unneeded implementation should those requirements prove inaccurate. Hmmm, if only there were some way to capture requirements as code without implementation details.... Oh, that's right. There is a way. And coincidentally, it's what we've been working on for the last six years here at GL.

Searching for love in all the wrong use cases

Having established (ie, forcefully asserted) that it is generally Bad to engage in ritualistic development practices, but Good to adhere to a particular practice when it's well-suited to surmounting some challenge, let's consider the first snarling beast that leaps out to menace most young, impressionable development projects before they've even secured funding -- requirements definition.

Functional requirements are strange and mysterious things. Many programmers have never actually seen any, relying instead on rumor, innuendo or hearsay to guide their development objectives. For many, functional requirements are the only things keeping programming from being their dream vocation. Programming after all is, at its best, a beautifully artistic endeavor wherein we devise clever structures to manage the complexity of some problem, and transform chaos into order. Implementation "elegance" is regarded as one of the highest achievements we can attain as developers. As each new project begins, we think wistfully of the various forms of abstraction and all the intricate frameworks we have mastered over the years, and consider how, armed with the depth of our experience, this project will surely be different from our past ones. This time we'll build the thing Right, methodically assembling a great edifice from a series of simple building blocks. Each will be as cohesive as the nucleus of an atom, but as loosely coupled from one another as the grains of sand along the seashore. You kick off your shoes, and wander along the beach, smelling the salt air and thinking about that time under the boardwalk -- then, suddenly, a subject matter expert comes marching down the beach with several unruly business owners in tow, and all hell breaks loose.

Real-world requirements are seldom elegant. On the contrary, they tend to be maddeningly messy. For many applications, the hand-waving of some business stakeholder-guy can be sufficient for reaching an understanding of what needs to be built. But, alas, the majority of applications we build are far more "interesting" than these. Confronted by the terrifying unknown of some obscure workflow that has evolved over millions of years and countless mergers and acquisitions, many developers immediately fall back on their religious texts for guidance and comfort against the onrushing horror. They begin frantically writing user stories, drawing anatomically inaccurate stick figures, and in the dark of the night wondering why the gods of order and logic have forsaken them despite their best attempts to follow the good and righteous path documented and indexed in their most venerated religous text (second edition, totally updated with a new preface by the author).

Why must it be so? Why do business requirements, which on the surface seem so simple compared to the complexity of the software platforms we, as accomplished software developers, whip into submission almost every day, so often defy our best efforts to wrestle them to the ground though we pelt them with CRC cards and lock them in use cases? Is there no way to reduce the arbitrary tangle of business rules to some elegant expression of the problem that can return us to our idyllic afternoon at the seaside?

Grab a towel and some sunscreen and allow me to introduce you to Precision Requirements Modeling.

Too Much Religion

Programmers are a religious sort, unquestioningly engaging in ritualistic practices pronounced as "good" by the industry's high priests, and likewise shunning those practices branded as blasphemous. Everyone knows for example that mixing business logic with user interface code will damn your soul to Hell for all of eternity. Likewise, engaging in waterfall development leads to nowhere but poverty and pestilence. And each time you use dependency injection, an angel gets his wings.

The religious tenets of our industry of course have mostly arisen from hard-learned lessons about what seems to generally work and what seems generally not to. Still, it is clearly the case that there is no single path to project nirvana. Every development initiative grapples with a unique variety of technical, political, and sociological challenges. Some projects even succeed. It is of course natural that people examine the practices used by the really successful ones, and strive to emulate them to maximize their own chances for salvation.

Successful software development, alas, cannot be achieved by blindly following the prescriptions laid down in some sacred textbook or blog posting. The sacred texts do indeed provide valuable techniques discovered only after the spilling of vast amounts of blood by our ancestors who came down out of their mainframes to spread out across the network, converting green screens to rich clients, and procedures to objects. So, let us embrace the tools and techniques, but let us also recognize that the reason software projects fail is that there is no "Way" that one can simply follow to guarantee success.

Software development projects present us with a series of challenges. Each of those challenges can generally be addressed with some established set of tools and techniques. When we complete a successful project we are typically tempted to look back on the steps we took along the way and announce to our colleagues, "Ah, yes, truly we have found The Way. Let The Way be written down so that it be made repeatably so, and we can look forward to nothing but happy days for ourselves, our managers, and our manager's managers." And yet we then see others try to follow The Way, only to find misery, chaos, and even death. Okay, maybe not death.

So are we condemned then to roll the dice each time we undertake a new development initiative? I don't think so. But the solution is not religion. The solution is in learning a few highly useful techniques, but rather than applying them ritualistically, we should instead learn to assess the particular challenges presented at a particular juncture of a particular project, and to select among and adapt the various techniques to the challenge at hand.

The Way does not exist. This is The Only Way.