Jason Lenhart

I recently completed reading JavaSpaces Principles, Patterns, and Practice and found it to be a quality book on the subject.  This was a great introduction to the concepts and implementation of Space Based Architectures. 

Coupled with the introduction are plenty of great examples, applicable to the Web 2.0 architecture being evangelized these days.

One thing that was really neat about this book was the foreword by David Gelernter, the father of Space Based Architectures.

Yesterday I gave a presentation on JavaSpaces within my organization - mainly to share some different approaches to implementing distributed systems around data.  Literally around the data, as JavaSpaces is all about processing units that associate (via a Template Pattern) into a network accessible object store.  The processing units can also be notified of data/state changes enabling a type of Event Driven Architecture.

One thing I keep coming back to in my mind is this concept of the internet working for people rather than people engaging the net.  For example, while not an exclusive Space Based Architecture; what was presented as Google Wave within the Google I/O conference fits the bill of data being shared by processing units looking for state changes.  The example they gave was set around email — mainly how email should be more of a collaboration of people acting on a virtual document.  This sounds a lot like an object in a tuple space ‘event-driving’ changes in state.  Reviewing the Google Wave architectural diagrams it is interesting to see how they are making concessions to deal with Partial Failures much the same way a Space Based Architecture would perform.

Overall, is it safe to assume that the “Next Big Thing” is to have the internet work for each of us in a personalized way?

For example, I would like to setup my own processing units around the web notifying me of health records changes, better insurance rates, and local restaurant deals, short-sellers killing my portfolio, etc…  — but to not go through umpteen different services (Geico, Fodors, and Fidelity).  Shouldn’t the web really just act as my own personal-grid (queue Depeche Mode here) that I can independently process to my hearts content (for free) in an event driven manner — who wants to pay for these services? — Certainly Not Me.  While social aspects on the internet are wonderful, perhaps it is time for all of us to use the internet much the same way that an Investment Bank determines Value at Risk.  Instead of risk - we can attempt to only determine Value.

The Google Grid???

I have been lucky to work with many talented individuals who have taught me so much.  Mark Little is one such person that I had the fortune to work with/for.  I found Mark to always go out of his way to help a developer or someone establishing open source communities (no matter his busy schedule).  It is of no surprise that Red Hat would ask him to take the top development role.  I wish nothing but the best for Mark and I know his experience and approach to developing quality teams and software will shine.  Congratulations Mark!

http://triangle.bizjournals.com/triangle/stories/2009/03/30/daily12.html

http://bobbickel.blogspot.com/2009/03/mark-little-jboss-cto.html

In the last couple of days I was asked to build a presentation regarding lessons learned, observations, and recommendations for the last project I technically directed and managed (doing this with an excellent group of peers I helped to recruit). In general, it got me thinking about SOA and the decisions we were making both successful and lessons learned.

First off, I could not help but think back to the fanfare of the project kick-off (some 3 years ago), the term SOA being utilized in every presentation. Whether these presentations were presented to the greater organization or within technical/non-technical conferences — the term SOA was certainly overloaded in an extreme manner.

As the project progressed, one thing that remained was the fact that technical leadership never lost site of the core principles of SOA - more importantly how it is nothing new or exotic. As a published friend of mine stated:

“Companies have been ‘doing SOA’ for many years, even before the term was coined, using technologies as diverse as CORBA and JMS. Think of it as a pattern, or an architectural approach in the same way as distributed object-oriented systems. It has its place in any good architects toolbelt and we’re finally coming to grips with it as an industry.”

I have a tremendous amount of respect for Mark Little - and have always felt his opinions are “spot-on” (to steal a quote from Ocean’s Thirteen).

So why are we just now seeing large studies that indicate that SOA is not working in most organizations? I mean - it seems so fundamental that Organizations would want to achieve certain key benefits (a sampling of the key benefits I dug up in past presentations, re-termed by me in parenthesis):

• Leverage high levels of interoperability to seize new business opportunities (fancy term for ability to build composite applications)
• Customize/leverage existing assets (not have to start from scratch on every project)
• Be able to leverage a best of breed landscape for software, data, and infrastructure (knowing that changes in the environment have been properly abstracted or loosely coupled such that these changes can occur with little to no impact in the consumers)

So why is it not working — who or what would impede these benefits from happening? Certainly there is a cultural aspect to this question which I am not going to address on this post but I will offer a more fundamental approach to the model or architectural style that SOA promotes, as there seems to be little question that the core ideas behind SOA are the right ones (they have been around for decades) — this approach, I believe to offer large organizations a better chance at success.

I think the issue with SOA is how so many organizations/people have gone about the design and implementation . The architecture community loves to review ‘reference implementations’, so what would a great example of a SOA design and implementation look like? I don’t think you really have to go far to realize it — most folks are using these implementations in their daily lives. Google with all the Web APIs which tap into Maps and Data, or Amazon and their successful Web Services, or how about the new social phenomena of Twitter that is getting 10 times more use through its APIs than its interface. These are all outstanding implementations of SOA. Furthermore, it is even harder to argue their success when you begin to look at all the “software mash-ups” currently out there in the Global SOA.

Getting back to my retrospective - how come we do not see this level of success in the Enterprise? We were seeing success - but shouldn’t the rest of the organization be sharing in it? I guess these are questions I was coming back to in my head as one of the things I would often hear:

“Well if we could start over - and use any technology we wanted - seemingly had an endless budget - we would have that success as well”

I don’t believe that only new implementations (with tons of money) can enjoy the fruits of an SOA implementation done right. There is a sort of implication in that statement that SOA is tied to the technology. I disagree - lets go into deeper detail.

Going back to my ‘reference implementations’ on the web, I am of the belief that organizations are often aligned in a way that is fundamentally unlike the web. IT organizations have long been plagued as a ‘cost center’ rather than a strategic asset. Looking at the above examples it is clear that there is a focus on consumption and openness as key essential characteristics to building an internal partner ecosystem/community. There certainly was not a big-bang approach to their strategy but it fostered a community of growth that is returning higher levels of ROI. The large IT organizations I have consulted/worked within are very effective for their perspective class of business problems (mainly providing a long term safety of corporate domain knowledge and systems) - but when SOA is being seemingly shouted from the mountain tops like a Ricola commercial - by its very hierarchical organizational make-up across various LOBs weighed against varying consumer dynamics, consumption and then taking into account the lack of easy to process data, forest of relational databases, and silos of proprietary applications — openness and consumption take a backseat to bonuses, perceptions, and plain ol’ ‘get it out the door’.

I believe proper design and implementation can allow these sort of internal ecosystems to develop independent of the organizational dynamics. What I have found is that by not always taking the ’standards approach’ has yielded good bit of success. Most notably in the approach and delivery technique to allow openness and consumption - not letting this be forgotten or ignored in favor of:

• Security Department rants, confusing openess consumption as an attack on security

• Groups/individuals spreading FUD about ‘if you are not using WS-* you will be refactoring your whole codebase very soon’ - thus freaking out stakeholders

The concept that has worked best has been best coined by Nick Gall as Web Oriented Architecture (WOA). REST/WOA is placing the focus on the resources and not on the services which is often where we see all the SOA methodologies taking the people who are performing the implementation

By organizations allowing and fostering their organizations to adopt WOA at a grassroots level - the openness and consumption will naturally progress. How can I make such a lofty assumption? - I have both seen first hand and in some cases felt the pain of how this direction was/would-have-been the better decision. This is especially true when you see companies make acquisitions and want to bring their business onto your platform, incrementally. Some observations:

• Instead of a few point services that typically have a context of usage that the LOB could only use due to constraints like WSDL (yes I believe one of the core tenets of loose-coupling in SOA is broken due to WSDL) - enterprise data can be tapped into through REST based resources (just like the web), which will allow any application to consume via HTTP and XML. Note: Even higher levels of access can be achieved via syndication which establishes the openness.

• Open Web APIs - exposing data to customers will often get LOBs out of the presentation domain. Data is king in the enterprise, why must everyone provide a front-end to every piece of data in the name of:

‘I cannot have you just willy-nilly accessing our data’ — so a front-end justifies this?

ACLs can be appied to Web APIs as well. It is important to note that openess is not a code-word for the lack of security - these are orthogonal to the implementation.

Next to an organization’s people, data is the most important asset - it should be managed like a money-manager manages a portfolio. Treating the data as restful resources will inherently offer productivity gains via external frameworks and tools. From a management perspective - I believe that it is easier to conceptualize the make-up much easier therefore people will not get caught up in a federation of relationships. Also, I observed that making the data a resource offers the opportunity to better engage you business with the types of tools they use everyday. Thus offering a medium of collaboration that helps get the business-partners out of the ‘IT is a Cost Center’ mentality. The business folks are writing code every day (whether IT knows it or not they do) — it is untested code and placed into production immediately. A quality example of this is the explosion in use of spreadsheets to perform higher level functions - whole departments will build out templates and email them around, using them to perform activities whereby they do not have to explain to IT (e.g. accounting). My point is that more so than ever before - we in the technology community are blessed (and I use this term loosely) with a business community that is operating a higher level of competence around technology - they can conceptually bridge ideas into the digital domain much more easily. That said, with tools like blogs, wikis, twitter, widgets, etc… this should enable better adoption of WOA because you are driving the consumption of the resources through the mainstream.

Overall my observations - are best summed up in the following:

Like previously stated SOA is an approach that the greater community has come to grips with, conceptually it has been around a long time - but the benefits will take much longer for the enterprise to realize (across the board) as the current enterprise techniques lack in grass-roots support. Some assumptions in the design of services and implementation of standards should not be so basic to the organization. Techniques that have the by-product of openness and consumption; using WOA, are the ones to promote and implement.

Every so often I find myself re-reading papers that have shifted my professional opinions.  One such paper is A Note on Distributing Computing which makes the fundamental case on why objects that interact in a distributed system need to be dealt with in ways intrinsically different than that of objects in the same address space (local object invocation confined to a single address space).  Such a great piece of work this paper is in my mind - they managed to keep their opinions very matter of fact and hard to disagree.  The vision of unified objects will never be realized - and not for the reason most like to cite, latency.

I often find myself cringing when I read a book or tutorial — see a presentation, and the statement is made:

“It is just an interface you don’t care where the implementation resides….” or “Remote is just like local…”

I have been there — I had those presentations whether on paper or in my head … then through implementation I learned about the wonderful world of “partial failure”.

When reading and thinking long about partial failure — I found myself going off on a tangent in my mind, especially when I reached this early statement:

“Partial failure requires that programs deal with indeterminacy.  When a local component fails, it is possible to know the state of the system that caused the failure and the state of the system after the failure.  No such determination can be made in the case of a distributed system.  Instead, the interfaces that are used for the communication must be designed in such a way that it is possible for the objects to react in a consistent way to possible partial failure.”

When you throw in the possibility of a resource manager providing derteministic state in a local component - I could not help but think of how a BPM suite can offer some level of comfort in partial failures.  In most BPM related applications there is the suite (holding the state of a long running transaction) and then there is the database (attributes of the long running transaction).  A partial failure offers a problem domain where they can get out of whack very quickly.  Why?  Because most large scale applications that utilize a BPM suite will make their business logic calls in a distributed context (the vendors would never want you putting your code into their environment and then providing support - not to mention scaling and partitioning issues).

The trade-off that I have made is to have a workflow process instance attempt to act as very simplistic (but deficient) external resource manager.  The interfaces are simple and provide a descriptive nature to allow the process manager to direct a partial failure to someone’s attention (manual interaction as partial failure should not be the norm).  For example, you might want to receive data into a system - a great opportunity to use a Pipes and Filters pattern:

•  Receive a proprietary format
• Validate format
• Normalize the format
• Enrich the format
• Persist the format

Now within a local context - I could receive the format and persist all in the transaction.  However, in the case of using ’services’ spread out over the network some webservices, queues, etc… I have an opportunity for a partial failure.  Now within the BPM suite I think this situation plays nicely into my hands.

If I have an issue with Enrichment of the data - a simple YES/NO Unit style interface can provide the “state of the system after the failure” such that the process instance can be routed to a repair state with meta-data describing the nature of the problem (particular elements in the format could not be enriched because …. environment, application domain, or business logic).  What is the trade-off?  Now my users have to understand the business workflow under the covers and how this flow is broken down on an invocation level (e.g. the data could not be persisted).  Otherwise the repair may not be evident.  You can offer better evidence on what a user needs to do via a user interface — but now we are mired down in additional solutions to problem domains presented to partial failure (a definite trade-off).

I think I have a strong argument around users taking on my technical understanding of partial failures - aren’t BPM suites heralded just for this sort of scenario.  They act as a medium of collaboration that bridge functional requirements to a technical specification.

Okay — so I told you this was a pretty big tangent.