Agile IT Architecture

Service Component Architecture

2009-05-20T16:25:00.001-07:00

Discussing about agile IT architecture on this blog without going to SCA will be a major error. I'm playing with SCA since last September, and I love it.
Service Component Architecture is defined as the implementation of the SOA architecture. You can read an excellent paper from David Chappell on this subject. SCA separates application business logic and the implementation details. It provides a model that defines interfaces, implementations, and references in a technology neutral way, letting us to bind these elements to any technology specific implementation.

For the business case and business user point of views the value propositions are around:
- Save time and money
- A simpler API, Efficient GUI tools to assemble components to build new application
- Enable and encourage reuse - Developers can create composites that perform useful functions. SCA makes it easy to use and reuse business logic.
- Bring agility to interchange business logic
- Bring visibility on how the application is built. I can imagine easily using the assembly diagrams to explain how 'business' components work together in the context of the current application.

On the architecture level the ability to separate business logic from infrastructure logic reduces the IT resources needed to build an enterprise application, and gives developers more time to work on solving a particular business problem rather than focusing on the details of which implementation technology to use.
Some key concepts:
A component implements some business logic exposed as one or more services that operate on business data. Component includes an implementation, and can have partner references and interfaces.

A component can indicate the services it relies on using references. Explicitly defining references allows dependency injection, the SCA runtime can locate the needed service and injects the reference into the component which needs it.
Components are bring together within an assembly or composite. The composite is persisted as an XML document using the Service Component Description Language (SCDL).

A component can also define one or more properties. Each property contains a value that can be read by that component from the SCDL configuration file when it’s instantiated. Finally a domain can contain one or more composites, each of which has components implemented in one or more processes running on one or more machines.

On the tools and implementation side, IBM for sure has an impressive offering on that space with WebSphere Integration Developer. Apache Tuscany is also an excellent open source to help us jumping into the technology and understand how it works. Eclipse Ganymede is offering an assembly editor, SCA project plugin...

We will build our AML application using SCA

AML with Event processing and Rule Engine

2009-05-13T10:08:00.000-07:00

I have to build a demo and a presentation (co -presented) for IBM WebSphere Impact 09 on how business rules and business event work together in the context of Anti Money Laundering. James Taylor did a good summary on his blog.
I promised some time ago on this blog to go through a complete example of executing ABRD on a project. So lets take this demo as a main example. Lets start with this first blog on a short description of AML and its high level process.

AML business context:

Money Laundering is the act of hiding illegally earned money from police and tax authority by making illicit funds appear as initiated from legal business. The Money laundering is a three step process. The first step, called ‘placement’, is done by depositing illicit funds in a business bank account. If one makes a cash deposit above 10000$ the bank is required to report the transaction to the government. The next step is called ‘layering’, wherein funds are moved from bank to bank and consolidated. The last step is the ‘integration’, where the funds are reintroduced to the financial system as ‘clean money’.

The first defense against money laundering is the requirement on financial intermediaries to know their customers— often termed KYC (Know Your Customer). Knowing his customers, financial intermediary will often be able to identify unusual or suspicious behaviors, including false identities, unusual transactions, changing behavior, or other indicators of laundering.
Placement rules should be able to detect deposit structuring by one or more individuals at various bank locations within a day or over time; notably this can also include ATMs. Large wire remittance customers, such as Money Service Business, will deposit cash more often and in greater volumes than typical customers.

Rules to detect large cash placements, using various methods and locations, in a single day, week, or month, would therefore be well-suited for monitoring.

Layering rules should identify bogus loans to offshore entities which never repaid the funds or the loan paid off in cash.

Current auditing happens manually where the auditor examines data output from legacy application and search for cash transactions over a period of time. The goal is to migrate to a continuous monitoring with software component which will alert auditors for suspicious activities. Banks need to be aware of all of the financial transactions that make up a ML. The critical knowledge to manage is the associations between such transactions.

The first step of a business process modeling approach is to work on the business process.

The high level process can be seen with 4 steps:

Detect fraud pattern
Analyze pattern
Investigate customer
Report on fraud

The detection of ML pattern is looking to different sources of information like the transactions, the customer accounts, the loan servicing applications, and looking at pattern of behaviors leading to potential Money Laundering. The detection of pattern is done with a time window constraint. A person making cash deposit on a yearly or monthly basis may not be a ML. A person doing cash deposit regularly without business motivation may be a fraudulent. The analysis is a sub process which aims to look at the potential fraudulent customer and search for historical information or customer data points already gathered by the system. Investigation is, as of today, a human activity performed once the system is reporting a risky customer. The investigation is to complete gathering information on the customer. Reporting Money Laundering to authority after the investigation is completed and positive.

From this process we will evaluate how to deploy event processing engine and rule engine for the pattern detection, analysis and investigation, in some next posts.

EDA and Rule Engine

2009-05-01T10:08:00.000-07:00

I presented sometime ago an architecture overview and use case for deploying a Rule Engine inside an Event Driven Architecture. There are papers from analysts and other bloggers on that subject which are predicting that EDA is becoming a hot subject in the next few months. I want to share what I found interesting.
Event Driven Architecture is an asynchronous publish-and-subscribe communication pattern: Publisher applications send events to a mediation layer which is notifying the subscribers interested by the events. The publisher is completely unaware of the subscriber. Components are loosely coupled in the sense that they only share the semantic of the message. The simplest Java implementation is based on using JMS Topics as it is a natural API for pub-subscribe messaging.
The data carried on the message payload are events with business means. The goal to embrace EDA is to deliver real-time access to business data. This is not really an extension of SOA but a complement of it as publishers may call services on event detection. But it can be seen orthogonal to SOA, as SOA is using a traditional procedural pattern around synchronous controlled orchestration of services.
Some are saying SOA is dead, replaced by EDA. Well SOA is still a valid approach to design IT architecture. SOA is not dead, and EDA is a complement of it. One thing that I think make EDA very attractive is the flexibility to add new function/application without impacting existing ones.
By the way EDA is not new: One of the mainframe programming models was to have batch application waiting for the result of other batches to process their own work. It is very close to subscribers waiting for events coming from publishers. At least we can say EDA is the distributed version of the old mainframe programming approach.

As soon as current application in the IT are able to post events, you have the flexibility to add/ remove listeners to address a new business needs.

So why rule engine is a critical component within EDA? One deployment is to use a BRE to support the implementation of such listener. Instead of developing rigid application you use BRE to bring the agility inside the flexibility. The component can be seen as a decision agent. The second interest is in the implementation of the event processing that has to detect event, process it and take action.

We can see multiple levels to support this event processing depending of the characteristics of the architecture and the type of event processing we look at. I'm seeing at least three:
- Simple event processing: the subscriber focuses on processing a few types of event with specific static conditions, and initiating action such as creating a new event, or calling a service. This processing can be real time or not. We may not want to consider any time dimension in the event.
- Event stream processing: Events are ordered and arrived as a stream to the subscribers. The processing may involve time windows, count based window, leveraging time based pattern,... It is used to synthesize data in real time.
- Complex Event Processing: Detect complex patterns of events, consisting of events that are widely distributed in time and location of occurrence. It supports low latency, high throughput, complex event management with aggregation, join, stateful operators, event A followed by B and by C, and all the combinations of it.
The technologies to support each processing are different, and it is important to do not use one for the other, or we will generate frustration.

Sustainable IT architecture

2009-03-05T08:24:00.000-08:00

In current difficult economic period where companies are merging, need to comply to new regulations, reduce cost, and be able to react more quickly to economic changes, building Agile and sustainable IT architecture is a must.
So the book "Sustainable IT architecture" examines the use of Service Oriented Architectures (SOA) from the perspective of its contribution to the development of sustainable and agile IT systems that are able to adapt to new technology developments and to manage business processes.
The book is the translation and adaptation of a first french book which has a huge success in the IT community in France, and other countries using french language. I contributed for one chapter of the book around BRMS and how this component is a major piece of the Agility Chain Management System. This book also arrived on a good timing, where some architects and journalists are questioning SOA, or claiming SOA is dying. There is no doubt about SOA, its values, and approch. SMABTP, an insurance company in France, deployed SOA starting 2002 and without it they will not have their current sane business. (Jean Michel Detavernier, co-autor of the book, did an excellent presentation at DIALOG09 on how he did deploy SOA within SMABTP). You should be able to order it soon, and do not forget to comment on this blog.

ABRD v1.2 available on EPF download

2009-02-27T15:53:00.001-08:00

Hi
To share with you the ABRD v1.2 is available for download as part of the EPF practices library, instead of dowloading it from CVS

http://www.eclipse.org/epf/downloads/praclib/praclib_downloads.php

Please continue to contribute and help on its enhancement. I thanks all the persons who gave me good feedback and propose changes that I did incorporate in current release or will do in the future release.
ABRD is becoming more and more adopted and used, it is a simple set of practices to develop business application using rule engine technology and so very relevant when we want to use BRMS platform.
I'm quite busy since these last months between the acquisition of ILOG and the book I'm writing on JRules and ABRD. I could not blog as I will expect.
But stay tuned, there are a lot to come in close months around BRMS ...

BPMN Modeling and reference guide

2009-02-09T20:02:00.000-08:00

As Derek Miers commented on one of my old blog, some of my BPM maps were not compliant to BPMN. He was for sure right, and I bought the book 'BPMN Modeling and reference guide' that I really encourage people to buy and read to be fluent around BPMN. I was short cutting the approach by looking at presentations you can find on the web. You can get your own interpretation of the notation and got completly wrong. So a reference is always preferable, and with this knowledge you may arrive to design good process, at least process compliant to the intent of this standard.
The book is a must to be sure every one understand what a BPMN diagram is representing. Each BPMN modeler tool on the market has his own interpretation or 'add-on' so it is always interesting to get back to the intent of the notation and the reference. The book is illustrated with a lot of samples. Very useful.
Thanks again Derek.

DIALOG 09- A Success

2009-02-09T16:55:00.000-08:00

Last week I was at Dialog 09, the ILOG customer conference. It was a real success, and real pleasure to see all the BRMS customers presenting how they are able to empower the business users to maintain rules. James is blogging on Dialog, and you can see good summaries there.

I was co-presenting effective rule writing, which is still a hot topic for business rule application, and there is some web seminar on this topic as soon as this week.
Another important subject is about rule deployment and the different strategies for deploying rules with the IT. I will blog on that soon.
The last presentation was on SMABTP, a very successful story on SOA adoption, and BRMS deployment: in 2002, Jean Michel Detavernier (CIO Deputy) has the Vision to embrace SOA at the enterprise level, deploy rules every where and put in place a real agile IT architecture. It is now possible to define new insurance product in days where it was needed months before. There are still customers, architects, and CIO who are questioning SOA value proposition, they may need to read such presentation. In such difficult economy, agile IT architecture is a Must. no question.

Manufacturing Equipment System and CEP

2008-12-17T16:23:00.000-08:00

I just finished a proof of concept for a manufacturing customer on how to use a inference engine to process events coming from a Manufacturing Equipment System. Without breaking any confidential information, I just want to highlight the use cases, the proposed architecture, and give some example of rules.
At the high level, the business use cases are real time fault detection and equipment monitoring. Equipments are tools running on the manufacturing floor. The manufacturing tools process parts. After a certain amount of work processed, a tool needs some maintenance and any parts assigned need to be inhibited so the MES can route these Work In Progress parts to equivalent tools.

Tool can generate alarms, and the system can take preventive action and/or alert people to avoid bigger problem. So for fault detection a rule looks like:

if Alarm_id == 36 then inhibit parts running on tool initiating the alarm, and send email and SMS message to floor manager

I have to complete this rule set with correlation, filter out and removing consequential alarms rules to avoid generating too much actions.

When {
alarm1: Alarm(category == TrackBlocked);
alarm2:Alarm(category == ScannerEmpty; equipmentId == alarm1.equipmentID; this != alarm1)
} then {
Retract alarm2;
}

For equipment monitoring, the manager wants to monitor limits of product to tool. Once thresholds are reached we can trigger a maintenance request on tool, and reroute parts scheduled to the tool. Events are sent at each job completion. Job means the processing of a manufactured part on a tool. At each event the rule engine updates counters, and verifies thresholds. If thresholds are met

The architecture has to integrate corba, and Java RMI.

The CEP server code is simple and uses JRules POJO rule session to insert events in working memory. The performance is to process 1 event per second, and manage a sliding window of one hour. (configurable). So the code is managing a sorted list of events and removes older events no more part of the time window. This is very simple and good enough for this simple event and time window management, as high volume and performance goals are far from what low latency CEP engines are used to process.

What is interesting on this project was the classification of the application as event based processing, and so entering into the CEP space, even if we are not really working on stream of events, or using any SQL like language or computing aggregation. This is a classical alarm filtering and correlation application where an inference engine can do a good job.

SOA slowing down

2008-11-21T14:32:00.000-08:00

Gartner says the "Number of Organizations Planning to Adopt SOA for the First Time Is Falling Dramatically". I will summarize the facts as below:

53% of person contacted are using SOA in some part of their organizations
25 % were not using it but had plans to do so in the next 12 months
16 % no plan to use it
20 % are building Event Driven Architecture
20 % are planning to do EDA in the next 12 months
Since the beginning of 2008, there has been a dramatic fall in the number of organizations that are planning to adopt SOA for the first time. down to 25 percent from 53 percent in 2007
Many organizations have evaluated SOA and have chosen not to spend time and effort on it
The highest concentrations of organizations not pursuing SOA and having no plans to do so are in process manufacturing and agriculture and mining
SOA adoption in Europe is nearly universal, moderate in North America and lagging in Asia

Main reasons listed are:

No clear business case – but there is a great deal of confusion about how to construct a business case for SOA
Lack of skill and no plan to acquire necessary skill
Use of modern programming environments is closely associated with SOA
Legacy skill + SOA are scared

In my recent work on the book "Resilient Information System" as co-author, I have to study what are the pains and use cases our BRMS customers have in term of SOA. I'm still doing interviews on that matter but basically there are four main subjects to address when migrating to SOA :

Data: work on the definition of domain data model, or put in place some transformation layer to present the different view of the data to the applications
Business rule: be able to understand, to externalize and to easily change the business rule outside of the applications.
Expose functions as business services: once reusable points have be identified, the services can be designed and documented (WSDL) for reuse.
Re-engineer the business processes to support automation using reusable services.

The successful deployments among our customers are done in that order. If CIO buys upfront the full SOA suite, he will most likely not get the bang for the money, as the migration to an Extended SOA is long. So the business case has to be built by increment with the first goal of deploying tools that empower the business user to change his business policies. When you are successful on this matter, the other business cases are easier to articulate, because business users are seeing the value. As explained in the Agility Chain Management System the core technologies to support the implementation are MDM, BRMS and BPM. Deploying SOA without BRMS and MDM does bring to the red path of the SOA maturity matrix. BPM, as BPEL orchestration engine, is not the solution to buy upfront, it will arrive later in the SOA maturity when we have reusable services that we can orchestrate.

SOA is a viable architecture design and approach as was OOD in programming, so please do not kill it. Now it has to be decided as a corporate initiative and implemented by increment, project after project.

BRE part of the service layer

2008-10-30T11:05:00.000-07:00

I just had a look to the presentation from Sandy Kemsley done at the business rule forum: "Mixing Rules and Process" and her blog entry on BPMS and BRMS integration. I can quickly summarize the points she made that I love and share some feedback:

Does BPM has Rules? yes but typically not full-features BR. Rule changes may require redeploying processes with IT involvement.
Separate rules from process: externalize decision from process. Call BRE from BPE.
Benefits from separation: compext rules automate manual process, reuse rule across processes, change rule without processes.

In her blog entry "As a BPM bigot, I see rules as just another part of the services layer... but I didn’t hear that from any of the vendors." It may be because ILOG team was not speaking... ;-)

We are pushing, saying and we are delivering projects since 4 to 5 years where we have a clear separation between decisions done by business rules from process flow, and designing solution where the business process is calling the decision service. In ABRD I even clearly propose this approach as a best practice for the reasons mentionned above:

different velocity of change and agility
reuse of decision point
better version control and change management support in BRMS

What we do observe in the field is that customers are at ease to design a web services or a services (SOA larger definition) that use a rule engine to implement the decision logic supported by these services, but are at pain to deploy executable processes, because not all the other parts (services) of the business process are ready. So BRMS is the first technology deployed in SOA rewampping before BPM. Green field deployment is difficult to find, and legacy application are moving slowly to be rewampped as reusable services. Not to mention the nightmare of data model management.
We are still at the lower left part of the SOA maturity adoption matrix and the Agility Chain Management.

JSR94 - an over visioned java standard?

2008-10-01T22:34:00.000-07:00

Recently I had to re-do some JSR94 code and I'm still interested by this work done some years ago and I'm still supportive of it. But as Roy Johnson was saying recently during one of his presentation "Where will tomorrow's enterprise innovation come from?". Does JSR94 is one of this "unhealthy Java standard" like JDO was? Done in a period where the Java community wanted to standardize everything ?
For the recall JSR-94 is an industry standard that defines how Java programs deployed in J2SE or J2EE can acquire and interact with a rule engine. Being able to change engine implementation is a nice design approach, but as of today this specification is limited by the fact that there is still no standard to exchange rule definition between engines. So rule written for one engine can not be used by another one. This dramatic limitation is undermining the use of JSR94. And i'm still surprise to see architect asking compliance to it.

Although the horizon is still blue: W3C is working on the final specification for Rule Interchange Format which should help to exchange executable rule definitions between engines supporting this specification. With RIF, JSR94 will make sense in the design of decision service.

SBVR thoughts

2008-09-12T16:29:00.000-07:00

I spent two days last week to study the Semantics of Business Vocabulary and business Rule or SBVR specification. SBVR is part of the OMG’s Model Driven Architecture (MDA) with the goal to capture specifications in natural language and represent them in formal logic so they can be machine-processed. It includes two specialized vocabularies:

* Vocabulary for Describing Business Vocabularies which deals with all kinds of terms and meanings.
* Vocabulary for Describing Business Rules which deals with the specification of the meaning of business rules, and builds on top of the previous vocabulary.

The meanings are declined into concepts, questions and propositions. The meaning is what someone intends to express or understands. A phrase such as "We deny the invoice if the medical treatment was done after one year of the accident" has a clear meaning for a claim processor within a car insurance company. Analysts need to transform logically this meaning into concepts which has a unique interpretation so that they can represent the business knowledge within a comprehensive vocabulary and rules.

Business rules are declined into two possible classes:

* Structural (definitional) business rule which are about how the business chooses to organize the things it deals with, they are considered as necessity. In this context the statements describing the rule can describe the necessity, the impossibility or the restricted possibility.
* Operative (behavior) business rule govern the conduct of business activity. They are considered as obligation and directly enforceable. When considering operative business rule it is important to look at the level of enforcement to specify the severity of action imposed by the rule in order to put or keep it in force. Statements to describe the rule include obligation, prohibition, and restricted permission.

In SBVR, rules are always constructed by applying necessity or obligation to fact types. Fact type is an association between two or more concepts.

SBVR is defining a complete grammar to specify business logic. I was thinking it may be too complex to get easily accepted by the field, even if it may been seen as mandatory when moving to an extended SOA: There is a need to define enterprise ontology, to avoid reinventing the definition of terms used by all the services deployed in the SOA.

From a project management perspective SBVR brings interesting risks as the project team members may engage in a huge effort of documentation, without delivering working software. Not so agile, this stuff. The just good enough should be kept in mind and a build by increment for this conceptual model may be a good approach.

The second important risk will be linked to the size of the universe we have to cover. Defining all the vocabulary and business policies for a given business application will generate a lot of facts and rules that may be implemented in different environments or not implemented at all. Then the decision to select which rule goes where, will add time to the project.

I had in mind to use a sub part of SBVR to help control the way we harvest rule during the discovery and analysis activities, but use it only in the context of what need to be implemented. The next ABRD drop for the end of the month will propose things around that.

Any comments are welcome.

HP printer and business rule

2008-09-04T17:13:00.000-07:00

AAAAAH. Today I discovered a very nice implementation of business rules done by HP. At home I have a HP photosmart 8250 printer who today displayed the message:"the ink cartridges are past its expiration date!"

So each ink cartridge has a date and the code in the printer is testing this business rule to force me to buy new cartridges.... It is not enough to get printer drinking cartridge ink like we drink water under 100F, but in case we are not printing enough, this rule enforces business.

Sad commercial behavior, bad rule, I'm just now hoping they are using a rule engine so I can change the rule on a fly... A dream.

IT and Business Analyst working together

2008-08-29T09:12:00.000-07:00

During one of my last web seminar I presented, I got the following question, I want to share the answer on the blog: "How much IT involvement would be necessary for making rule changes, assuming that the business person can change the rule and test it on their own?"

IT is always responsible of the production platform. As such it is controlling the quality of the rules deployed to the servers. As the simplest process you can have the business users maintaining the rules within the web based BRMS component and even deploy to a test server to do some simulation testing and what-if analysis. Once the rule execution set is ready for deployment to production, IT can create baseline, extract the rule set from the rule repository and then apply some non-regression testing suite to verify that the rules are still working as expected. This is a good timing also to verify that any changes done at the data model level (logical or physical models) are not impacting existing rules.

The following diagram illustrates the different activities per role, I can easily imagines as a standard maintenance process:

So IT is still responsible of production quality control, version control, software components integrity, production server monitoring and management and nothing will change in close future.

CEP and BRE

2008-08-22T10:36:00.000-07:00

I spent the last four months to dig into Complex Event Processing offerings and especially how BRMS and CEP can work together to support new type of business applications, and to move to agile IT architecture. As stated by Professor David Luckham in his book the ‘Power of events’: future applications need to address the following problems

Monitor events at every level of the IT systems
Detect complex patterns of events, consisting of events that are widely distributed in time and location of occurrence
Trace causal relationship between events in real time.
Take appropriate action when patterns of events are detected.

Adding CEP engine within our IT architecture will help bring agility as we can define complex patterns of events, hot deploy them, execute them, and improve their scope by time. CEP solution today uses a language close to SQL to define statement which filters, aggregates, joins events, and applies pattern matching on stream of events. Those statements are deployed to a CEP engine which will continuously evaluate them on a flow of events, when the conditions match subscriber applications (or code) will receive the complex or synthetic event and do something about it. If we consider those statements as rules, BRMS is a good candidate to manage them as it offers all the efficient tooling to consider those rules as asset with their own life cycle.

As of today BRMS uses Rete based engine to evaluate and fire if-then-else type of rule. An event processing statement looks more as a query. For example if we want to extract bid events on given car brand, event occurring in a time window of 15 seconds and we want to compute the average price (one of the attribute of the BidEvent). The statement may look like:

select *, avg(price) as avgprice

from BidEvent(itemName='BMW').win:time(15 seconds)

I’m using the Esper open source product EPL syntax to illustrate this example. This is a very simple ‘rule’ with the expressiveness of SQL. Easy to understand for a programmer but with big limitations when we need to communicate with the business user on what rule is doing. This is more true in real application. Most of the time those statements will become more complex to understand (even for programmer) when such statements combines joins, aggregation, multiple streams and database lookup. Putting a high level language on top of this SQL based approach will help on that matter. JRules for example offers a framework called BRLDF to define business language on top of lower programming language.

Those statements have a business motivation and CEP applications are really pushed by business users as those technologies are helping to identify pattern of events relevant to the business. We are moving to real time BI. So with this business dimension we can consider CEP statements as business rules. This means BRMS can support the management of those statements, and offers an integrated environment for business analysts and developers.

Now, one of the main questions is the deployment model. Does Rete based engine support well the high throughput and low latency of current CEP application requirements? The response is very close to no, if we speak about millions of events per second, rules to apply on sliding or jumping time window… If the constraint on the number of events decreases BRE may be a good solution. We are using JRules or Rule C++ in the telecoms industry since mid 90s for alarm filtering and correlation applications and it was the main driver of the BRE demand at that time. So we can use Rete for event processing. In fact in long term we can imagine having different engines under the same product or one engine which will pick up the more efficient algorithm to use according to its deployment model. I will post some example of rule using event in a next post.

Example of a rule analysis

2008-07-20T09:20:00.000-07:00

I will try to do some specific rule analysis to illustrate some of the ABRD concepts. ABRD proposes to do the analysis activities as soon as possible and in parallel of the rule discovery, this helps to identify the issues, gaps, and clarify the rules under scope. So let imagine a claim processing application for a car accident. There are some injured persons who reported the accident claim and with some medical providers who send medical bills to the insurance company. One important step of the rule discovery is to clearly understand the business process. For this example we can outline the basic process for claim processing looks like the following diagram:

In brown the decision points that are rule rich, so will be mapped to a rule set. Taking one rule from the adjudication decision point we will build a domain model:

“Ask for an expert audit if one of the treatments includes an emergency room treatment or an ambulance transfer done on a day after the accident. “

Claims adjudication in insurance refers to the determination of a member's payment, or financial responsibility versus the company ones. So obviously the company wants to pay for what it commits too no more. Here the policy states that we do not want to pay for emergency visit or ambulance transportation which was not on the same day of the accident. For the emergency room we can have some exception, if the injured person came back two days later for headache, or pains that could be consequences of the accident, we need to get some complement of information. For ambulance there is mostly no reason to pay. So from this little analysis we need to develop two rules: one for the emergency room case and one for the ambulance as the actions are different. One action will trigger some audit request and the other action will set the reimbursement amount to zero.

The analysis process starts by looking to the business terms used by the rule. Here we can highlight: audit, treatment, day of accident (also referenced as day of loss) and type of treatment like emergency room, ambulance transfer. Let starts by the fact that the claim is related to a car accident, this means our object model should support a claim type with a day of loss concept attached to it. There is at least one injured person who needs medical treatment, and so the claim relates to the involved persons in the accident. Injured person has injury and related treatments. A medical bill includes the description of the medical treatment done on the patient. By discussing with the claim processor or google ‘medical procedure code’ we can understand that each treatment has a code that we can use to get the type of treatment. The medical bill in fact has some legal form like the UB92 or HCFA-1450. From those forms we can also build our data model.

The audit may include a list of reasons so that we can manage in one audit multiple questions or items to review. The claim and the medical bill will follow a life cycle which will be supported by a state machine. Part of the state machine management will be done by the rule, part by the application. For example when rules are finding issue on the claim they will change its status. So from these descriptions we can build the following UML class diagram

Once we have this UML diagram, let generate Java classes, load them in a java project within Eclipse and tune this model so it will be more efficient from writing rules, using a test driven development approach and refactoring. I encourage using Maven to package the project and to be able to efficiently manage jar dependencies in your project. Once we did so we can start JRules Rule Studio and create the Business Object Model entry. I will detail that in a next blog.

Very good success with recent webseminars

2008-07-18T14:00:00.000-07:00

Hi
Thank to all of you to attempt my last web seminars on ABRD, SOA, and Effective rules writing. You were around 850 to register and close to 400 to attempt. This is showing an interest in best practices around business rules and their implementation. Feel free to ask me more questions by email, and I will start to respond on this blog and on the isis blog to the more common questions.

SOA and BRMS

2008-06-12T12:06:00.000-07:00

Well I did not post since a while. I was working on an interesting subject related to Complex Event Processing. I will detail that in close future.

In the mean time coming back to our purchase Order sample, I will take the architect's hat, and outline the deployment of BRMS within a SOA approach.

SOA has proven to be a great technology to make enterprises more agile. With their ability to separate business logic from its implementation, BRMS adds more capabilities to the SOA. SOA is a progressive architectural style for creating and using business tasks that are packaged as services. The main goal of these services is to expose loose coupled business functions to facilitate deployment, combination, and reuse within different applications. SOA supports the integration of heterogeneous systems, as soon as they are exposed over the internet using standard protocols like HTTP, XML, unified interface definition (WSDL).

Every business has rules, and business users want complete control and visibility over these rules. A BRMS decouples an application's business logic from its data access and from its flow control. Decision logic is exposed behind decision services which support some business task. Such services are ValidateCustomer, ValidatePurchaseOrder, CheckCreditHistory… Using BRMS those service delegates the processing to a rule set deployed within a Rule Engine. Those rule engine can be managed using a rule execution server which provides the pooling mechanism, rule set management and execution monitoring functions.

A business application leveraging MDM, BPEL and BRMS may look like

This generic diagram can be used to explain how BRMS can be deployed in SOA. In dark blue the components coming with ILOG BRMS.

The SOA is organized in a set of business services used by the business application logic. This business application layer is modeled using use cases diagram or business process map, and may be implemented using BPEL orchestration, and/or a standard Model View Control web application. The business services leverage lower level services, which I packaged in a technical service layer. They support lower granularity service like accessing business objects (DAO for Creating Reading Updating Deleting data), master data like product definition, enumerated domain value, sub process flows, web services and rule execution server. In this logical view of the architecture I do not need to expose the Enterprise Service Bus which may need to be used as a communication layer between the application logic and the business services.

In fact this diagram even if it outlines one application, can be applied for multiple applications. The business and technical services can be reused to support other business cases.

Rule developer creates the rule projects structure, the data model used by the rules, the rule execution flow, prototypes some rules, and then gives the hand to the business analysts to complete the rule authoring activity. Business Analysts use the web based interface to manage the full life cycle of the rule. Rules are persisted in a rule repository. The deployment of a rule set can be done without stopping the core application, and can be controlled within the Monitoring and Control environment of the production platform. Using a BRMS the IT staff and business analyst can work closer together as they are using the same high level language to write business rule, they understand the rule flow, so the long term maintenance of the application is greatly enhance.

Within a SOA the BRMS is bringing an effective and efficient mechanism to manage the decision logic of any business application. The architect needs to design the decision service so they can be reusable, then the rule architect needs to design the rule execution flow to take into account some specificities of context of execution. Rule set reuse may infer some tests on the context of the caller. This context is exposed to the rule engine as a fact in the working memory, and the rule flow can have some initial tasks to test and apply different branching according to the state of this context.

I will dig into code detail later.

A sample to illustrate the ABRD approach

2008-05-11T16:01:00.000-07:00

There are a lot of requests from colleagues, customers, readers of ABRD to get a sample covering the main activities of ABRD from A to Z. I'm working on a book with a friend of me, which will cover all the ABRD activities in detail, and provide detailed samples. But for the moment I will take a simple purchase order process. There are a lot of samples on the web around this process, but here I'm focusing on a simple version of it.
This Order Process may betypical B2B scenario, in which multiple stateful services interact with each other. Two sources for the order request: one online with an Internet customer submitting an order on a Web page, and one using an automatic purchase order issued by a well known authenticated customer-partner.

In ABRD using a business process centric approach for discovery – analysis – implementation we start by doing the analysis of the process, then we go to the data modeling, rule harvesting, and service design in parallel like the following diagram outlines it:

During the process modeling the rule and process analysts search for verbs which involve mental processing or thinking like check, qualify, compute, calculate, estimate, evaluate, determine, assess, compare, verify, validate, confirm, decide, diagnose.. to identify activity which has business decision and logic, applies some business knowledge... Those activities should be what we call "decision rich" and have the focus of the rule analyst to discover rule.

Purchase Order Process description

The process is initiated when a customer online submits a product order, or when an automatic purchase order issued by a well known authenticated customer is received on one EDI link. A customer validation step is executed using customer data and product request data. Based on the product data provided in the order, it is checked whether the product is on stock. If the product is available, the price for the shipment of the ordered product is determined and presented back to the customer. If the product is not available, a message is sent back to the customer by email or web page (if still online). This last case also triggers email to product manager to provision the stock for this product.

The total price of the order is calculated from the product price, the customer business condition, loyalty program… and the shipment price. When the charging and the shipment have been performed successfully, the order is filed and the process ends.

Process Analysis

Looking on this process and the description above we can extract the following decision point where business rule will apply:

Decision Point Name	Description
Validate Customer data	Verify the customer information – data present or not, and black listed customers
Validate Purchase Order	Verify there is no fraud in the purchase orders or on a set of purchase order coming over a time period, , verify the data are correct
Calculate Total Price	As the price may account for a loyalty program, marketing campaign, stock level, and other business dimensions, business rules are relevant here to compute the price outside of a simple addition.
Get Shipment report	It may be possible to receive a set of events from the shipment company to aggregate and correlate to the same purchase order so that customer can be notified on a major type of events as well as the internal management team.

A decision point table is an important deliverable of the analysis. It can be built during the inception phase by the team responsible to outline the project high level scope. If it is not present the project team has to quickly develop it during the first iteration of the project.

We will detail the outcome of the rule discovery in a next post.

Successful webseminar on ABRD

2008-05-11T15:52:00.000-07:00

Hi
Thanks a lot, to all the attendees on last week web seminar , we got records of registered persons and connected ones. There are a lot of questions, and I could not respond to all of them. So I will use the BLOG to answer the questions related to Agile Business Rule Development and other architecture discussions.
For Europe we plan to do a second webseminar, I will announce it.

SOA maturity Matrix, Agile Chain Management System

2008-04-25T12:00:00.000-07:00

Today I would like to present a very nice article in a IT french journal “Solutions logiciels”. This is related to sustainable architecture. I’m working with Pierre Bonnet, the author of this article, on this matter, and try to share experiences on BRMS, as one of the corner stones of the Agility Chain Management System, I will try to summarize the main points of this article in french as it is relevant over the world:

The IT infrastructure in a lot of companies suffers from a lack of investment to modernize the legacy, which adds complexity when development team deploys new applications. Business and IT managers are focusing in short term ROI putting on hold long term investments needed to build a long term sustainable architecture.

Added to these, we can foreseen three layers of IT staff: the senior with their knowledge of the current legacy systems and the business requirements, the intermediate layer with people skilled in client/server and EAI technologies, and the younger one with object oriented skill, and web, web services development backgrounds. Younger are involved in tactical projects, using agile methodology, but most of the time generate rigid systems not able to change and supporting a long term logical architecture. The seniors are starting to retreat leaving a deep gap in knowledge which is most of the time not transferred to younger population. These are future bombs in the IT organization. What to do?

IT management needs to start long term projects to rebuild the IT architecture for the next generation. Pierre proposes to start by looking at what is the current situation at the enterprise level and being able to build a plan to transform the IT architecture for long term needs. The second point is to take risk: we can manage an IT organization just by controlling the existing. The rapid changes in economy, new regulations, and technology progresses force managers to take risky choices. But the risk is also to look at change in IT staff population, skill transfer, and risk to deploy new application on a fragile architecture.

Finally his last point is about clearly communicating to business and executive management about the non quality contract of current IT system, the medium to long term degradation of the IT services, and so the company competitive advantage. The communication has to leverage a strong action plan leveraging a risk management plan.

Three principles support the argumentation:

1- The re-engineering of any IT architecture cannot being done in one shot. The diagram below presents a progressing path to migrate existing architecture to a sustainable one. Starting from a SOA to expose the existing legacy functions as services, then moving to the extended SOA where BPM, BRMS and MDM are used in conjunction to offer agility at the service level for change in referential data, decision logic, and business process. The end of the path references the use of those technologies for the enterprise architecture.

2- The Agility Chain Management System: Using the MDM, BRMS and BPM technologies, IT staff can start by externalize referential data and parameters using the MDM, business rules and decision logic using BRMS, and orchestrate services and support business process with BPM.

3- A Enterprise Level Methodology: the migration path enforces a strong use of methodology like TOGAF, Praxeme, CEISAR)

Pierre addresses in this article some very important points and principles to take care during a SOA migration. Currently we are seeing a lot of project focusing on re-vamping legacy function as web services but this is not the end of the story of a SOA. I like very much the path to the extended SOA and the maturity matrix as a tool to explain to IT management the long term vision. Project managers can position where their project stands on this matrix to clearly articulate what is the value of their project’s product for the long term IT vision.

I modified slightly the original maturity matrix to clearly mention that any project falling into the bottom-right corner is the beginning of decision service re-engineering using the MDM-BRMS-BPM technologies, that IT development team has to leverage for the long run. When people embrace those technologies they are seeing the value and how to leverage them at the enterprise level. As you can see green path is the way to go and the red path is going no-where as it falls back into a rigid application behind a web service...

As an example of this kind of project we can take a claim processing application where the development team and architect decided to migrate one of their service (Claim Coverage Verification) to use business rules to verify the coverage of the insured person around his claim. MDM was used to define all the referential data as the medical codes, the US states, zip code,.. and other application parameters. The legacy application offers some data and business services with a re-vamped web service approach, and the BPM was used to call the services, to dispatch the set of XML documents over the process steps, and to offer the GUI for the claim processors: the actors of the business.

Starting from there the team can enhance the migration of such application and design reusable, agile technical and business services.

Book your agenda: May 7 10.00 am PST. Web seminar on ABRD

2008-04-22T13:37:00.000-07:00

Make Developing Business Rule Applications Easier

What: 60-minute best practices webinar
When: Wednesday, May 7
Who should attend: Architects, developers, application programmers, software engineers, IT senior managers, IT team leaders, and anybody who wants to learn more about developing business-rule applications

Learn about a rules-specific methodology designed for short turnaround times: Agile Business Rule Development (ABRD)

When you're building business rule applications, generic software-development methodologies just aren't enough.

Agile Business Rule Development (ABRD) is a step-by-step process for developing business rule applications. An iterative methodology, ABRD employs agile software-development values. Rule development is organized as a series of cycles: discovery, analysis, authoring, validation, and deployment. Your team stays on schedule, delivering outstanding business rule applications.

Attend this free one-hour webinar and learn how to:

Leverage ABRD as first open-source methodology for business rules
Implement and deploy rules in an SOA or BPM context
Engage in rule discovery and analysis

Software Development Life Cycle for BR-BPEL application

2008-04-02T16:14:00.000-07:00

The purpose of this article is to present an integrated software life cycle for development team who wants to leverage the capabilities offered by technologies such as BPM-BPEL and Business rules management system.

Each new business application development is triggered by a business needs to support or enhance and improve business efficiency. Even with an Agile approach, to trigger the project, architect, business users, project manager and project sponsor have to work together during the Inception phase, to define the business case, the requirements and justifications of the project. Once the project funded, the team may work on specification and requirement documents. I simplify a little here because in reality a lot more documents may be produced, but the goal here is to present how those requirements are supported in an Agile way using the BRMS and BPM products and not discussing about requirements management. So let state the specifications are our main entry point to detail the major work that needs to be done for developing the applications. The format of the specification can be user stories, or a more detailed description. (a good reference to write specifications and user case is Alistair Cockburn work ). Specifications are classically managed by a standard SDLC which includes design, build, test, and deploy of working code to the different staging platforms (development, test, or production).

The Agile approach enforces short iterations to deliver quick value to the business. The blue tasks in the diagram below represent a set of iterations that build the core of the business application.

The horizontal axe of this diagram represents time. The pools are used to contain activities executed in the different development environment. The top pool is the business analyst team which is triggering change to the application. When the development team is using BPEL and BRMS technologies, there are parts of the specifications which go directly to those components: Business processes will be implemented using BPEL and BPM process map, and business rules using BRMS. In the first release (you can see a release as a vertical slice of the above diagram) the agile value of each of these technologies is not clearly articulate. This is true that BPEL and BRMS help to develop in less time, and adopt an agile approach of development. But the main value proposition will come in longer term by the ability to efficiently support the two types of change requests the business user may ask:
• The business rules and policies changes: business want to improve the decision done by the rule engine
• The process tuning or improvements: the process needs to change due to legal constraints, better error and exception path management, or for better execution.

Business Rule changes will be done in the BRMS using a web based interface or IDE like JRules Rule Studio. The hot deployment capability of a rule set authorizes to deploy a newly updated rule set in a question of seconds. This helps a lot to support quick change on the decision logic attached to decision services called by the BPEL process instance. Rule Developer can do a lot of try and catch in a pre-production environment to improve the quality of the rule set, so the quality of the decision. Some IT organization implements the “champion – challenger” pattern to evaluate a new rule set against the production one. The champion rule set is in production and process transaction request with a certain level of metrics: metric can be number of automatic decision, or other business key performance indicator. On the pre-production server, the challenger rule set receives the same data flow, and metrics are compared. If challenger beats the champion it becomes the new champion in production.

Business process improvement requests are managed using the BPMS system, and a new process definition can be released to the different platforms. We can definitively see more change requests coming for business rules change, than on the business process: When we have a working process it is more sensitive to change it. Still business process changes will happen to support for example an improvement of the exception management, or to find some short-cut on the global processing so that the time to complete the process is better.

The core application can still evolve with a new release path. Change to the GUI, or data model will most likely be managed by a new release of the full application.
As a conclusion it is important to consider using an agile development methodology which support quick change when using BPM – BPEL – BRMS technologies, if you do not adapt your way to develop business application, you may lose the opportunity to be agile.

Industry First Open Source Methodology for Business Rules

2008-03-20T10:53:00.000-07:00

Yesterday ILOG Inc announced his donation of his Rule Based Methodology to Eclipse Consortium. I need to provide some explanations on what this donation is about. The Agile Business Rules Development methodology (ABRD) is the industry’s first free, vendor-neutral methodology delivered as an Eclipse Process Framework (EPF)OpenUp plug-in. ABRD provides a step-by-step process for developing business applications using technologies such as Business Rule Management System, BPM, BPEL.

ABRD mitigates the risk associated with new business rules initiatives by providing a well documented and structured approach for developing rule-based applications. ABRD allows organizations to avoid using ad-hoc processes or having to expend significant time and effort creating their own best practices.

In case you never have a look at EPF, Eclipse Process Framework provides tools for software process engineering to develop methodology. It comes with content knowledge organized in library, and with a tool, EPF Composer, which enables process engineers and managers to implement, deploy, and maintain processes for organizations or individual projects based on the content of the library.

The goal for EPF is to deliver a platform for producing software development practices, how-to, common definition and vocabulary, and processes with task, role, work product and guidelines definitions. Libraries are physical containers for knowledge content, process configuration and other parameters to publish the content as a set of web pages. Method content describes what is to be produced, the necessary skills required and the step-by-step explanations describing how specific development goals are achieved.
Processes describe the development life cycle. They take the method content elements and relate them into semi-ordered sequences that are customized to specific types of projects. They express who, when, what work will be performed.
EPF Composer provides a knowledge base of intellectual capital which we can browse, manage and deploy. This knowledge base organized in plug-in forms the basis for developing processes. We can define Roles, Tasks, Work Products, and Guidances in a hierarchy of folders named Content Packages. All content can be published to html and deployed to Web servers for distributed usage. Finally process engineers and project managers can select, tailor, and rapidly assemble processes for their concrete development projects. Once content is defined you can define the process building blocks, called capability patterns, which represent best development practices for specific disciplines, technologies, or management styles.

'abrd_openup" is a plug-in which extends openUp. From there you can reuse the content to develop your own plug-in. I recommend to extend ABRD without modifying it, so that you can leverage future release of it. If you want to contribute to abrd_openup you can send me email or comment on this blog. I will integrate most of the contribution in each release of the plug-in.
Have EPF fun!.

Agile Business Rule Development EPF plugin.

2008-02-27T17:21:00.000-08:00

Eclipse Process Framework just integrated the Agile Business Rule Development plugin into the OpenUp library. This abrd_openup plugin addresses the development of business application using rule engine technology and Business Rule Management System. This is pre-release 1.0 but it already integrates a lot of content around rule discovery and analysis. I am working on the architecture track and will be able to deliver a new version in March. So any comments or contributions are welcome.
Please download EPF and the OpenUp library and let me know your thoughts.