mue - embrace change

To be continued

2009-04-01T22:44:00.003+02:00

In case you're worrying that the series about Erlang/OTP won't be continued be assured, that I'll add more content sure. But beside several other tasks I'm currently making some progress in migrating my root server as well as other web locations on two new servers - cheaper but also better. And here I'll also consolidate the web presentation, the old wiki, and this blog. I'll notify you soon.

Software Architecture with Erlang/OTP - Part 7 - Supervision Trees

2009-03-19T23:00:00.002+01:00

In some parts Erlang is like the real life: there are many processes which do the real work and some which are supervising them. And there's a hierarchy. As I said, real life. *smile*

Typically Erlang/OTP applications are seperated into worker processes, which are doing their jobs, and supervisor processes, which monitor the worker processes. They get notified in case of a terminating worker. Beside the process id the reason for the termination is part of this signal. This way it is easy to implement a fault tolerant behaviour where a supervisor is able to restart an abnomally terminating worker process he's responsible for. Because supervisors are just processes too they can also be supervised. That's needed to build modular systems with a clear separation of concerns.

Supervision trees can be build manually through linking processes with the function link(Pid) or when spawning it with spawn_link(Fun). Normal terminating processes will cause no special reaction, but abnormal terminating processes lead to the termination of all linked processes. This may already help in many situations, but there are more possibilities. Calling process_flag(trap_exit, true) inside the process that should act as a supervisor leads to transformation of signals into regular messages on the format {'EXIT', Pid, Reason}. Now it's possible to receive these messages and react, e.g. through restarting the terminated process.

That's a standard situation. So it's no wonder that the OTP contains a module for this job. And the module name supervisor doesn't wonder too. *smile* Like the generic server and the generic finite state machine the supervisor needs a callback module. This time it only has to provide one function called init(Args). The result is a restart strategy, values to define the restart frequeny, and a list of child specifications. The list of the supervised childs can also be changed through adding and deleting child specifications at runtime. Each child specification is a tuple containing an identificator, the function to start with module, function name, and arguments, a restart strategy, a shutdown strategy, a flag if the process is a worker or itself a supervisor, and a list of depending modules for the release handler. This list already shows that there is much room to configure a supervisor.

So for example the Tideland EAS supervisor is starting and stopping channels with


%% @spec (atom(), atom(), pid()) -> {ok, Pid} | {error, Reason}
%% @doc Start of a channel.

start_channel(EasModule, ChannelId) ->
    StartFunc = {EasModule, start_supervised, [ChannelId]},
    ChildSpec = {{channel, ChannelId}, StartFunc, transient, 10000, worker, [EasModule]},
    supervisor:start_child(?MODULE, ChildSpec).

%% @spec (atom()) -> ok | {error, Reason}
%% @doc Stop of a channel.

stop_channel(ChannelId) ->
    supervisor:terminate_child(?MODULE, {channel, ChannelId}),
    supervisor:delete_child(?MODULE, {channel, ChannelId}).

The exact configuration is very well documented. But there's one last question. How to maintain the state of a restarting process? You already know the comfortable way the gen_server provides. But the state gets lost if the server process restarts. So in this case it's a common pattern to use a special second process, an ets table, or a Mnesia ram only table. But this overhead is only needed in case of process states that can't start at zero. If server processes work stateless and e.g. use a database as backend it can be kept simple.

Software Architecture with Erlang/OTP - Part 6 - gen_fsm

2009-03-17T23:15:00.001+01:00

Beside the generic server gen_server one of the most practical modules for own processes is the finite state machine gen_fsm. It is very similar to gen_server regarding the implementation of the user functionallity inside a callback module and its options for tracing, logging, and statistics, timeouts and hibernation, clean terminations and the handling of code changes. This makes it once again comfortable to concentrate on implementing the business logic without hassling with many runtime aspects.

In addtion to the standard behaviour callback functions of the finite state machine, take a look at the documentation, the developer has to implement additional functions for each state. Think about a process building a bridge between the two external systems foo and bar. First both shall be connected using connector processes specified through special modules, then the connector processes poll them and notify the bridge for transfering the data after a conversion. If one of both signals a stopping the bridge shall end its work.


init({FooModule, BarModule}) ->
    {ok, FooPid} = FooModule:start(FooCfg, self()),
    {ok, BarPid} = BarModule:start(BarCfg, self()),
    {ok, awaiting_connection, {FooModule, false, BarModule, false}}.

awaiting_connection({connected, foo}, {FooModule, false, BarModule, false}) ->
    {ok, awaiting_connection, {FooModule, true, BarModule, false}};
awaiting_connection({connected, foo}, {FooModule, false, BarModule, true}) ->
    FooModule:poll(),
    BarModule:poll(),
    {ok, polling, {FooModule, BarModule}};
awaiting_connection({connected, bar}, {FooModule, false, BarModule, false}) ->
    {ok, awaiting_connection, {FooModule, false, BarModule, true}};
awaiting_connection({connected, bar}, {FooModule, true, BarModule, false}) ->
    FooModule:poll(),
    BarModule:poll(),
    {ok, polling, {FooModule, BarModule}}.

polling({received, foo, Data}, {FooModule, BarModule}) ->
    BarModule:send(foo2bar(Data)),
    {ok, polling, {FooModule, BarModule}};
polling({received, bar, Data}, {FooModule, BarModule}) ->
    FooModule:send(bar2foo(Data)),
    {ok, polling, {FooModule, BarModule}};
polling(stop, {FooModule, BarModule}) ->
    FooModule:stop(),
    BarModule:stop(),
    {stop, normal, {FooModule, BarModule}}.

This is a trivial example, but it already shows one of the great strength of gen_fsm. It is really simple in Erlang to use concurrent running processes for different tasks, e.g. using the generic server. But if you do only synchronous calls there's no win opposite to sequential programming. When using asynchronous cast messages instead it's difficult to handle the also asynchonously resent results. And here the finite state machine helps a lot. A business process is a kind of state machine too. So it can be implemented using the gen_fsm module. Parallel activities can be started once and the results collected before continuing with the next step.

As already mentioned above the finite state machine knows event timeouts like the generic server. But sometimes a business process has an overall maximum runtime. So it is possible to start a timer with gen_fsm:start_timer(Time, Msg) at any time and tell it to send the defined message inside a timeout event when the time is over. Depending on the current state this event can be handled inside the callback module.

The combination of gen_server and gen_fsm is like the combination of services and business processes. Both work very good together. How they are supervised at runtime I'll describe in the next entry. So, stay tuned!

Software Architecture with Erlang/OTP - Part 5 - gen_server

2009-03-16T22:30:00.004+01:00

The next entries I'll cover the most important modules of the OTP, today the generic server behaviour gen_server. It is the base module for server processes with a standard set of functions and functionallity. This allows to concentrate on implementing the business functions while inherit many useful features like tracing, logging, and statistics, synchronous asynchronous communication, management of process states, and handling of code changes.

Take for example a little counter process. It is implemented as a callback module:


-module(counter).
-behaviour(gen_server).
 
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).
 
init({InitalCounter, Step}) ->
    {ok, {InitialCounter, InitialCounter, Step}}.
 
handle_call(get, From, {Counter, InitialCounter, Step}) ->
    {reply, Counter, {Counter, InitialCounter, Step}}.
 
handle_cast(count, {Counter, InitialCounter, Step}) ->
    {noreply, {Counter + Step, InitialCounter, Step}};
handle_cast(reset, {Counter, InitialCounter, Step}) ->
    {noreply, Counter, {InitialCounter, InitialCounter, Step}};
handle_cast(stop, {Counter, InitialCounter, Step}) ->
    {stop, normal, {Counter, InitialCounter, Step}).
 
 handle_info(Info, State) ->
    {noreply, State}.
 
terminate(Reason, State) ->
    ok.
 
code_change(OldVsn, State, Extra) ->
    {ok, State}.

It will be instantiated and used with


{ok, Pid} = gen_server:start(counter, {0, 5}, []),
 
gen_server:cast(Pid, count),
gen_server:cast(Pid, reset),
gen_server:cast(Pid, count),
gen_server:cast(Pid, count),
 
Result = gen_server:call(Pid, get), % Returns 10.
 
gen_server:cast(Pid, stop).

Instead of calling the gen_server functions directly they are typically wrapped. This hides the mechanism and makes it more user/developer friendly and maintainable. So far, so good, but nothing special. But it gets more interesting with all the additional features of the gen_server. So it is possible to add a local or a global name of the process so that processes can be addressed more easy. All using processes can directly send messages to the registered process. And it's possible to send message to locally registered processes on multiple node parallely. Other start arguments control how long the initialization may last before it is cancelled or if the process shall be observed through the sys module. So events can be logged or trigger user defined higher order functions. Additionally system events can be traced and statistical informations retrieved.

Until now I've only described the communication on demand. But sometimes it may be interesting that a process reacts on idle time. E.g. when the process has to work for a few minutes each hour but the other time the usage of resouces shall be reduced. Here the gen_server provides to interesting features. So if all messge answers also contain a timeout value it will be taken to send a special message to itself after the according idle time.


handle_call(Msg, From, State) ->
    % Also return a timeout value in milliseconds.
    {reply, Value, NewState, Timeout}.

handle_info(timeout, State) ->
    % React on the timeout, e.g. hibernate the process.
    {noreply, NewState, hibernate}.

Additionally the timeout can be used otherwise, e.g. for background calculations, checking caches, or lazy writings. And also the hibernation can be done on demand through return hibernate as part of a handle_call / handle_cast return.

It's easy to see how flexible the gen_server is. But it also has some drawbacks. When doing synchronous calls there may be a dependency between process like A calls B, B calls C, and C calls A. Then, without a specified timeout, a deadlock happens. With a timeout this can be prevented, but the behaviour is not the wanted one. I'll show later in this series how this situation can be handled. Another problem may be registered processes. Here only one instance exists per node or global. So it may be a bottleneck when heavily used. In this case only a kind of broker should be registered and dispatch the messages to special worker processes. But in this case you've got realize a special way for the process state. And this solution should not be a bottleneck, too.

You'll find more informations in the Erlang documentation, tomorrow I'll cover gen_fsm. So, stay tuned!

Software Architecture with Erlang/OTP - Part 4 - The Language

2009-03-15T19:30:00.004+01:00

First of all, don't expect an Erlang tutorial here. If this is needed I recommend the book "Programming Erlang: Software for a concurrent word" by Joe Armstrong, the inventor of Erlang. But I want to highlight some special features of Erlang that have a great influence on the design of many modules and libraries, and so naturally also on the architecture of the Tideland EAS.

Erlang is a functional language, not as pure as e.g. Scheme, but still functional. The arity of a function may differ, also the types of the arguments due to the dynamic typing. Pattern matching and guards, which I'll describe below, help to handle this flexibility in a very powerful manner. Additionally Erlang functions allow tail recursive calls. This is very important because the language doesn't know loops. Instead recursive functions are used, even for endless loops which are used in processes. In this case there will be no stack overflow, even if those still can be produced. *smile* The organization of functions is done using modules, public functions have to be exported explicitely. From other modules they are called using my_module:add(1, 2). The nice fact here is that both parts of this call, the module and the function name, can be replaced with variables. This is a very useful feature for the implementation of callback modules. They allow to change the behaviour of functions. For example take the processing of some data where you want to be flexible in the how it is persisted.


persistent_process(Id, PersistencyHandler) ->
    OriginalData = PersistencyHandler:load(Id),
    ProcessedData = process(OriginalData),
    PersistencyHandler:store(Id, ProcessedData).

Now different modules can be implemented with exported functions load/1 and store/2, e.g. one using the dets module, one using the Mnesia database, or any own implementation. They have to be passed in the function call:


persistent_process(4711, dets_persistency).

Modules like supervisor and gen_server use this mechanism called behaviour. The finite state machine gen_fsm even manages also the current state inside a variable and calls the according callback function:


dispatch({'$gen_event', Event}, Mod, StateName, StateData) ->
    Mod:StateName(Event, StateData);

This way the implementation of functional and non-functional features can easily be separated, generic modules build the fundament for reusage.

Two of the features I've already mentioned above and that I don't want to miss are pattern matching and guards. They are used to choose one of a set of different implementations of the same function depending on the passed argument. They work the same way when choosing the right actions inside of receive, case, and if statements.


calc(discount, Amount, AnnualTurnover) when AnnualTurnover > 5000 ->
    {ok, Amount * 0.9};
calc(discount, Amount, AnnualTurnover) when AnnualTurnover > 1000 ->
    {ok, Amount * 0.95};
calc(discount, Amount, AnnualTurnover)  ->
    {ok, Amount};
calc(comission, Amount, AnnualTurnover) when Amount > 500 ->
    {ok, Amount * 0.05};
calc(comission, Amount, AnnualTurnover) ->
    {ok, Amount * 0.02};
calc(Any, _Amount, _AnnualTurnover) ->
    {error, {unknown_operation, Any}}.

So when called as calc(discount, 100.0, 2750.0) the result is the tuple {ok, 95.0}, and the call of calc(comission, 100.0, 2750.0) leads to {ok, 2.0}. Any other operation than discount or comission leads to an error. While there are many situations where this feature is helpful it especially helps when developing a library with behaviours:


dispatch(Action, From, Handler, State) ->
    case Handler:process(Action, State) of
        {reply, Data, NewState} ->
            From ! {ok, Data},
            {ok, NewState};
        {ok, NewState} ->
            {ok, NewState};
        {error, Reason} ->
            From ! {error, Reason},
            stop
    end.

When developing a handler module the function process/2 has to be implemented with different patterns for the action:


process({add, Principal, Description}, State) -> ...;
process({set_credential, Principal, Credential}, State) -> ...;
process({deactivate, Principal}, State) -> ...;
process({activate, Principal}, State) -> ...;
process({authenticate, Principal, Credential}, State) -> ...;

Here we see another nice feature, tuples. It's very often needed to pass or return more than one value. But it is inconvenient to define special records or classes each time. In Erlang it is very simple to create tuples and to retrieve values out of them using pattern matching. Tuples and lists together allow real powerful data structures.

So, what's left? Processes, the most important feature of Erlang. The are lightweighted in starting, stopping, and at runtime. And they are isolated, what's extreme important for the reliability of Erlang applications. A process can just to a one time job and terminates when it is finished. Or it can be a longer running process responsible for different tasks, see the authentication example. But how does it know what to do when it is isolated? Each process has an own message queue and when starting a function as a new process a process id will be returned. This id, or a registrated alias name, can be used as the target for a message sending. Messages can be received using the receive statement and pattern matching. So a process loop function may be:


loop(Handler, State, Timeout) ->
    receive
        {From, Action} ->
            case dispatch(Action, From, Handler, State) of
                {ok, NewState} -> loop(Handler, NewState, Timeout);
                stop -> stop
            end;
        stop ->
            stop
    after
        Timeout ->
            case Handler:timeout(State) of
                {ok, NewState} -> loop(Handler, NewState, Timeout);
                stop -> stop
            end
    end.

The recursive called loop statements inside the function let it run endless until it is stopped explicitely or as the result out of the dispatch function. Now a message can be sent through


Pid ! {self(), {add, MyPrincipal, MyDescription}}.

The self statement passes the process id of the calling process. This is needed to resend the result. A process never handles two messages at the same time, they are processed one after the other. Statements like that are very often encapsulated in public functions of the process module:


Result = authentication:add(Pid, MyPrincipal, MyDescription).

The function also contains the receive statement to get the result from the process. Annother feature you can see above is the timeout inside the receive. When there's no message received inside the defined timespan in milliseconds the following statements will be executed. That's helpful for automation and needed for a non-blocking process flow.

I hope these first impressions of Erlang do help. The next blog entries will be shorter and cover the major OTP modules which are based on the above mentioned features. Enjoy it.

Software Architecture with Erlang/OTP - Part 3 - EAS Overview

2009-03-13T13:57:00.004+01:00

The architecture of the Tideland EAS is mainly organized around four major modules:

The bus is the one face to the user (developer, administrator) of the EAS. There's no real bus, but one process per node managing the supervisor, the registry, and the configuration. Additionally this module provides functions for starting and stopping of bus, channel, and service, for subscribing and unsubscribing, for the start of business processes, and for the raising of events.

The channel is used by the bus. It just manages its own service subscriptions and passes the received events to those services. There's one instance running per channel. It does no filtering of the content, this has to be done by the services themselves, e.g. using pattern matching and guards.

The service manages it's callback module and the service status. The latter is set only once during the initialization of the service. This is because the real work is done inside a process ring buffer provided by the Tideland CEL. So multiple requests can be handled concurrently. If real states are needed the service itself has to care, e.g. through the usage of the ets module. There's one instance of the service process running per service.

And at last there's the business process manager. This one is different from the three modules above because an instance will be created for each new process and it is terminating when the process ends. Internally it is implemented as a state machine and also the callback modules which are implementing the real business process are working as state machines.

Beside those modules there are some additional helpers. But the bus, the channel, the service, and the business process manager are the most important ones and the way how they use and communicate to each other is the core of the EAS.

Before describing how they are implemented and why I've choosen this way I would like to give a very little introduction into the language Erlang - no real tutorial, just the most important features that are also responsible for the EAS architecture - and into the major modules of the OTP. This will be the content of the next blog entries, so stay tuned!

Software Architecture with Erlang/OTP - Part 2 - Motivation

2009-03-12T12:58:00.004+01:00

When starting a private project it's helpful to have a vision in mind. In my case there are two projects I'm carrying with me for a longer time: the first one is a mixture of a semantic wiki and a blog, the second one a special community portal I'll surely write about later. Meanwhile there's a vague third idea, but it's too early to tell more about it. *smile* Beside those projects I want to learn more about Erlang, about how to do concurrency-oriented programming - I'm doing OOP now for more than 20 years - and how to realize an event-driven architecture.

So I've started with my typical bottom-up approach, first developing a library with useful stuff (the Tideland Common Erlang Library), then the server (Tideland Events and Services), and as the last preparing step, some always useful services (Tideland Common EAS Services). Those three projects build the foundation for the real projects mentioned above.

The major part surely is the EAS. The major design goals for this software are:

Following the event-driven paradigm of raising events in different sources (front ends, business processes, adapter to external systems, and more) and processing them in subscribed services.

Multiple subscribers should be able to handle the same event.

Supporting a loose coupling between and a high cohesion services.

Flexible assignment of callback modules to services and subscription of services to channels through configuration.

Using typical erlang mechanisms like behaviours for an easy implementation of services and business processes.

Technical aspects like distribution and pooling are transparent for the developer of services abd business processes.

Simple API, pure Erlang, and no XML for configuration. *smile*

The EAS tries not to be the next great application server, and it doesn't want to be an even greater integration platform. But I want it to be a solid and flexible foundation for scaleable Erlang applications on the server.

So next time I'll write about the general software architecture and the modules.

Software Architecture with Erlang/OTP - Part 1 - Preface

2009-03-11T19:12:00.001+01:00

During the last months I've developed the base for my future Erlang applications, the Tideland Events and Services (EAS) server. It's not yet finished, but I've learned a lot about how to use the concurrency in Erlang, pitfalls, tricks for stability and maintainability, and several of the standard Erlang modules. There are still some blank spots, but I'm exploring them more and more.

So I decided to publish a little series of articles about software architecture with Erlang/OTP here in my blog. It's not a real tutorial, but I would like to share the experiences I made and why they led to some of my design decisions. Starting point is my motivation for the development of the EAS, then an overview of the major OTP modules and how I'm using them, where to use concurrency, ways to handle the control flow in an concurrent environment, robustness and exception handling, concepts for flexibility and simplicity, working with functional domains, persistency with Mnesia, and the deployment of Erlang applications. What's still open is the topic web frameworks. Here I'm still not really sure how I'll do it. There are two or three ideas in my head, all with pros and cons. We'll see.

Currently I don't know how granular this series will be, how deep, and how log. This surely will also depend on your feedback, I appreciate it. So, stay tuned!

2009 whisky tasting season started

2009-02-14T16:30:00.010+01:00

Yesterday the 2009 whisky tasting season started at Kottkamps. I've really looked foreward to it and it has been nice to see our community again. This year there will be some more events due to the part-time shop Sandra and Ulli opened. So the next tasting together with Scottish cheese, salmon, and chocolate will be on Feb, 27th.

This time I've first choosen a Benriach 12y Jamaica Rum due to the Rum finish of one of my favorite malts, the Glenfiddich 21y Havanna Reserve. First it hasn't been so good, with raisins, but too muddy and mouldy. It became better over the time, but not really good. For the second one I've choosen an Edradour 1996 11y Rum, once again a chance for a malt with a Rum finish. It has definitely been better, for an Edradour very mild in nose and flavour, warm and smooth, and with no peat and smoke in the long lasting finish.

It has been a fine selection, but topped through number three, a Caol Ila 1995 Distillers Edition. This malt will also be part of my collection in future. A sweet and peaty nose, tastes of spices, malt, and smoke, and a long, warm, complex finish. That's how I like it. To close the evening I've choosen another Islay, the Laphroaig 2000 7y. Nedless to say anything, a classical and wild Islay, powerful. A wonderful finishing.

On OOP, COP and the Tideland EAS

2009-02-05T20:12:00.001+01:00

My current software project is the Tideland EAS (Events and Services) as an Erlang/OTP based platfrom for my future server-side applications. To understand the design motivation behind the EAS it may be useful to get a little insight into my software development experiences.

I'm now doing software development since about 25 years, the last 20 years the object-oriented way. So my development style is mostly characterized by my personal understanding of OOP. The bundling of attributes and behaviour is a natural way of thinking and easy to understand unless the imperative way of programming isn't etched on the memory too hard. Depending on the experience of the developer it's more or less simple to determine the right candidates for pure helper classes, for those building the foundation for the own system, and those representing the business model of an application. A large number of design patterns helps to find solutions based on a common understanding.

One of the most important parts of object-orientation is the idea of message passing between instances. When developing Smalltalk Alan Kay has been heavily inspired by Carl Hewitt's research on the actor system for massively parallel, distributed, computer systems. Unfortunately instances in languages like Smalltalk, Java, or C# aren't massively parallel. So messages sent to an instance block the caller until the work is finished. *sigh*

Think about a restaurant, where the guests have to wait for their waiter who is then waiting for the cook. The cook is then cooking the meal, one part after the other, hands it out to the waiter who is then handing it out to the guests. During this the waiter can't attend on other guests. So the restaurant would need a pool of waiters to handle every guest parallely. But additionally it would also need enough cooks to serve all orders.

To allow an environment where at least such a concurrency with pools is possible the traditional environments use threads. While those are relative lightweight from the operating systems point of view they are still very heavy on the instuction level. Sadly threads are not native for objects, they are orthogonal, a kind of meta level. So instances can talk to other instances in different threads without any problem while they are working on another task. Semaphores and synchronized blocks shall help here, but the more complex a system gets and the more a development team size grows the easier it is to confuse the system, to let it get more unstable. No good basis for reliable systems. Here I'm thinking about our waiter from the example above. Guests and cooks are calling for him while he is trying to satisfy all in parallel. It's easy to imagine how fast this poor guy gets confused.

Here I come to the concurrency oriented programming, actor systems, and Erlang. While in traditional OOP languages the default behaviour is synchronous and asynchronicity has to be added manually or through special frameworks it's different in Erlang. Function calls are also snchronous, but the exchange of messages between concurrent running processes is asynchronous. Blocking calls can be simulated through an immediate waiting for the reply sent back from the process. The OTP provides functions for this inside the standard modules. Looked at that way COP seems to be very close to the original idea of OOP. In case of Erlang this gets combined with its functional paradigm, transparent distributed environments, immutable variables, no shared data, pattern matching, guards, and the integrated database Mnesia. And even if the syntax seems to be weird for the curly braced fraction it's due to the Prolog roots more clean than the actual Java or C# standards - even though Smalltalk and Scheme are still more elegant and compelling. *smile*

Those very positive characteristics made me chosing Erlang as the base for my own development. But I want a bit more than the standard OTP provides, the reason for me to develope the EAS. The name Events and Services already unveils the idea of services implementing the functionality of business domains and communication between them in the form of events. So internally the EAS is a publish/subscribe architecture. The services subscribe to topics on channels and front ends, adapters, and services publish their business events together with a topic on a channel. Some events will be handled totally by one subscriber, others will be a flow of events, serial and parallel, between multiple services. Even typical calls will be just a flow of events. The initial event may be new_customer together with the customer data, the answering event could be customer_added together with the processed data. The publisher of this event may also be a different service than the initial receiver of the first event.

The first benefit of this approach is an extreme loose coupling - good for maintenance, very good for extensibility, and even better for scalability. But there's also a second aspect that's driving my design. Typically only one class or component is receiving a request, publish/subscribe architectures allow multiple receivers, so does the EAS. First this is interesting for automatic concurrent activities. E.g. each time a contract gets cancelled the system parallely updates the customer file, tells the financial accounting to strike a balance, and notifis the customer about the receiving of the cancellation. Or inside an order process the event order_ready_for_fulfilment could start the shipping and the generating of the invoice which do their work independently. But by far more interesting is the Complex Event Processing (CEP) with different types of analysis. Events may be checked that they always occur in pairs, with a specified rate inside a time span or a time period, their order is in a specified sequence, they happen in combination, or they are matching specific conditional expressions. Based on rules those conditions may be associated and correlated in domains, their evaluation leads to the generation of new events for an automatic reaction, alerts may be rised early due to detected signs of risks, it leads to a non-technical monitoring of the environment, the understanding of the dynamic nature of the business may be improved.

I've used the subjunctive here a lot, consciously, because the benefit isn't for free. First detecting the right business domains for the implementation as services and to not build a too fine granularity isn't easy. And second detecting the right correlation domains together with the optimal ruleset is a hard job and one has to think hard and long if it fits for the own situation. I'll see which experiences I'll make when the systems are up and running. But it definitely it's interesting and worth a try.

Building a process ring buffer

2009-01-27T22:16:00.000+01:00

During the last days - beside finishing my next Erlang article - I've made some thoughts about a fast process buffer for the usage inside my EAS platform. It should work almost like gen_server with synchronous and asynchronous messages and a functionallity implementation inside a callback module. The first attempt has been a dynamic pool based on a list of processes and able to grow and shrink. It already worked fine, but somehow it didn't satisfied me. Too complicated, too blown, and still the dispatcher as bottleneck.

So the second attempt is based on a ring buffer with the ability of multiple dispatchers. It's not yet implemented,but the concept already contains most of the code. The major challenges are (a) the handling of unexpected terminating processes and (b) how to allow the processes in the buffer to share a state. The first one is solved through linking each worker process in the ring to the next one. Additionally not only the pid of the next process is stored. Each worker process also keeps the pid of the process after the next one. If the next one fails during the handling of a message a new worker will be started with the pid of its next one as argument. During the restart the new process links the next one and retrieves the pid of its successor. This way the broken ring is closed again.

As to the second challenge I decided to differ from gen_server. The init function of the callback module just returns one static initial state. Inside the process ring buffer there's no change of this state. If such a behaviour is needed the callback module has to setup an ets table or something similar and use this. An intensive usage may indeed lead into a new bottleneck.

Following the concept there's now at first one initial dispatcher creating the ring. If it receives a call or cast message it will call its actual worker and receive the next one from it. The called worker casts the message to itself and is so able to handle it asynchronously. The dispatcher immediately can receive the next message and dispatch it to the next worker. If needed more dispatchers can be spawned and connected to the ring in a defined distance.

I'll report more one I've got my first test results.

SOA India 2008 slides available

2009-01-08T23:18:00.002+01:00

The slides of my presentations at the SOA India 2008 last November in Bangalore are now available online. They are

On Event-Driven Architecture and

Agility And The Way To SOA.

Have fun. Now I'm just waiting for the recorded movies. It will surely be interesting to see oneself presenting, at least it would help me to improve myself.

No 17 and 18

2009-01-07T22:38:00.002+01:00

My malt collection has grown by two new bottles. The first one is a The Arran Malt Sherry Cask Finished, a fine single cask (cask no 32 with bottle 444 of 581) with 54,8%. Even with this cask strength it can be drunk without water. The nose is sweet, with sherry, berries, and fine spicy notes. The taste is very powerful and voluminous with notes of pepper. It pleases with long and warm finish, once again spicy and without peat and smoke. My team gave it to me as a parting gift. Thanks again.

The second one has been a christmas present I made myself. It's a Glenrothes Vintage 1991, 15 years old and with 43%. The nose is mild, fresh, with fruites, vanilla, and butterscotch. There're also some notes of moss and spices. It tastes smooth, once again with berries, vanilla, butterscotch, and some notes of sherry. The finish is warm, long, very pleasing with no sharpness. A great malt, one of my new favorites.

Erlang process pool development

2009-01-07T22:21:00.004+01:00

During the development of my EAS I encountered a problem regarding the services. The first implementation had only one process for each service. Due to the allowed synchronous requests the first problem could be the locking I've described yesterday. OK, the EAS always uses timeouts and the logic is intended to be developed as the pure publishing of events on channels which will be handled asynchronously, but it still has been dissatisfying. The second problem could be the overload of single service processes which do not scale on multiple cores or processors.

The resolution shall be to introduce service pools. A master manages this pool and distributes the event messages on worker processes. And because this may be useful in other projects too I decided to let this process pool be part of the CEL. Additionally it shall be a dynamic pool, starting with a configurable minimum number of processes, growing on demand up to a maximum number, and shrinking when a worker process is idling long enough. If all workers are busy the message will be sent back to the end of the queue and processed later. The functionallity of the worker processes has to be developed in callback modules. They are almost like those for gen_server.

The first implementation has been a naive approach, with a master process for the pool and the message management and a set of worker processes. The creation of new workers has been done one by one. Everything worked fine, fast, and smooth as long as the initial pool size has been large enough or the messages are synchronous. But when starting with a small pool and sending asynchronous messages those have been received by the master process too fast. The resending of messages and the the returning of free workers back into the pool has been too much. So when the master process has been stopped not all messages have been processed and it lasts a long time until the stop message reached the process. Really funny, but not as desired.

So I created a third process, the buffer, responsible for the creation of smaller worker pools en bloc, the management of the whole workers, and the receiving of free workers. The master now only manages a smaller local buffer and retrieves a new one from the buffer process. His main job is to receive the user messages and to distribute them to the workers. First tests have shown a by far better behaviour. The whole system is now faster than before, especially the min/max/asynchronous configuration. First tests with 5 million requests and 500,000 worker processes are no problem. Now it's time for stress tests with multiple message sending processes, synchronous and asynchronous, and with random timings.

The source can be found at Google Code in the celsp*.erl modules.

Great job guys

2009-01-07T18:31:00.001+01:00

As I wrote in December I'm now working in a different team customizing and maintaining a commercial application based on Smalltalk and Gemstone/S. This month a new colleague started. Currently he has no experience in Smalltalk. So I've choosen Pharo as an environment to teach him, also because the GLASS team seems to test Pharo for future realeases.

I'm really pleasantly surprised. Even if it's still the version 0.1 it looks professional with it's OS X like look'n'feel - there's also one for Vista. And the integrated tools and the reorganization of the menus support the work with Pharo. It's real fun and definitely the right direction.

So once again, great job guys.

Avoiding Erlang deadlocks

2009-01-06T18:33:00.001+01:00

A friend of mine - a long time and very good Java developer - mentioned in his blog that Erlang can produce deadlocks too. In his example a process A send a message to process B and waits for a reply. During the processing of the message in process B this one sends a message to process A and waits for a reply too. But this message won't be processed due to the waiting of process A. And indeed, even if this example is trivial, he's right. The situation may for example be more complex, with a circle of several dozen processes all sending synchronous requests until the lock occurs.

But the model behind Erlang isn't the traditional request/response way like it is done between instances in OOP. It's more like a communication between colleagues in a bureau all busy doing their stuff. Sometimes one has a question. He may ask his co-worker and wait for an answer, and waits, and waits, and waits. But at some point it will be enough and he will react, through asking somebody else, through asking again, or through throwing a book at his colleague. *smile* That's like using a timeout in a receive statement, or the default behaviour of gen_server:call(). No deadlock, no endless waiting like a dumb sheep - hmm, even those won't wait very long.

Another way would just continue doing the own work until the answer comes. This is highly asynchronous and needs to split the logic into small pieces and to remember the question in the state of the process. This way processes could juggle multiple tasks. Using OTP with supervisor trees and gen_fsm for state machines helps a lot here. Additionally if needed all those concurrent working processes may share a process with a pure getting and setting functionallity for common status data or they may use one or more ets tables.

So, once again, yes it is possible to produce deadlocks in Erlang. But the way to produce it is an improper way of developing in Erlang. Using receive with timeout or using the OTP default timout behaviour - no calls with the argument infinity - is the first step, create solutions using the concurrency right the second one.

Another year goes by

2008-12-22T20:44:00.007+01:00

Only two days until Christmas Eve, once again the right time to recapitulate the passing year. The most important part of it has been the renovation of our house. It has been a lot of work (and also a lot of costs *sigh*), but it has been worth it. Especially our new kitchen is great, we love it.

Another important part of 2008 has been the SOA India. The preparation, the travel, the time in India, the sessions, everything has been exciting. I really thank Lynton for giving the hint to Axel who invited me. He took notice about my Erlang activities here in my blog. And that takes me to the next point, my Erlang developments. I've switched from Smalltalk to Erlang because the concept behind Erlang is clear and interesting for my favorite domain, the implementation of server-side applications. I will use it for the realization of a small Event-Driven Architecture. Annyhow I will not leave Smalltalk. The syntax and the clear object-oriented system are wonderful, the development environment is extreme powerful. After my job change, and that's the last big item, I've got the luck of using it professionally. Even if this hasn't been the reason for my resignation it is a very nice side effect. But more important is that I'm now more satisfied with my work and the results I see after the end of the work day.

So far so good. Enough said about the last year, now it's time to look forward.

May all your wishes for 2009 come true.

Job changes

2008-12-20T23:05:00.000+01:00

Some times it needs some time until you realize that you're on the wrong way. And this applies not only for driving, it's also true for the job. Mine has bee to manage a team of software developers for two products. But the more the team growed and the less I've had to do with software technology the more uncomfortable I felt. Not good. But my condition improved during the preparing of my sessions for the SOA Days 2008 (by the way, the slides are online). So I decided to enter my resignation as team manager together with the wish to continue my work as a senior consultant on a more technological level. Thankfully my employer has been able to follow my ideas and found a very good new job for me.

In my new team I'm responsible, together with my colleagues, to maintain and enhance an application for the configuration management of large telecommunication networks. Today the main part of this system is based on VisualWorks and Gemstone/S. Yes, you're reading right, it's Smalltalk together with an ODBMS! And the next generation uses GLASS. So it's fun, technologically, functionally, my role, and the team.

I immediately refreshed my Smalltalk and tried to port some of the Erlang/OTP principles to it. It's nice to have processes with message queues like in Erlang. With the idea of a publish/subscribe architecture in mind I also found a nice solution for the waiting for an undefined number of replies in a given time. And this idea will now find its way into my Tideland EAS. I think my work with both plattforms will have positive influences on each other.

Thoughts about an alternative Erlang syntax

2008-12-03T21:36:00.005+01:00

Due to the shift from processors with higher frequencies to systems with multiple processors, cores, and hardware threads functional languages and especially Erlang gain higher attention. Unfortunately the special syntax based on the Prolog roots of Erlang scares many potentially interested developers. I personally can adopt new syntaxes relative fast, during the time I've developed in different languages like Pascal, Java, Python, Smalltalk, and Scheme. So it hasn't be a great problem to learn the Erlang syntax. The greater challenge is to think in concurrent working processes with asynchronous messages. But that's not the topic today.

The question is, how an alternative syntax for Erlang could look like to reach a greater audience? I already wrote here about doing OOP with Erlang. But that's not the best approach, it doesn't change the syntax. Today I think an alternate notation would be better. So lets try with a quick example as a basis for discussion. But before some preconditions:

The designed language will not match Erlangs full feature richness, it's more a best-of-breed building some of the gen_server features direct into the language.

Also it's no OO-language, it's concurrency-oriented with asynchronous messages between processes.

It tries to be as functional as possible.

A code example:


module my_module {

    /* Define a simple function. */

    public function add(A1, A2) {
        /* Each statements ends with a semicolon,
           the last state determines the reply,
           variables are only assignable once. */

        A1 + A1;
    }

    /* Define a process. */

    public process my_process {
        /* Properties are addressable in each message
           function, but they can only be changed once
           there. */

        property Name;
        property SubProcess;

        /* Init will be called automaticly when a
           process is spawned. */

        public message init() {
            Name = none;
            SubProcess = spawn_linked(another_process);
        }

        /* The number of spawn arguments can vary. The
           adequate message function will be found through
           matching. */

        public message init(NewName) {
            Name = NewName;
            SubProcess = spawn_linked(another_process, Name);
          }

        /* Message functions use pattern matching and guards.
           In this example, the sender will be notified
           asynchronously. */

        message set_name(NewName) when is_list(NewName) {
            /* A dot is for synchronous messages. */

            case (SubProcess.analyse_name(NewName))
                ok : {
                         Name = NewName;

                         /* sender is predefined like self,
                            it's the sender of the currently
                            processed message. */

                         sender->name_set(NewName);
                     }
                _ : {
                        /* An arrow is for asynchronous
                           messages. */

                        sender->invalid_name(NewName);
                    }
            }
        }
    }
}

So far, so good. But some aspects are still open. First I would like to add hash maps. They are a powerful instrument for good readable code. So a message function could look like


public message withdraw(Booking) when Booking[amount] > 1000.0 {
    case (self.get_balance(Booking[account]) > 1000.0) {
        true : {
                   self->withdraw_checked(Booking);
                   ok;
               }
        false : {
                    {error, balance_too_low};
                }
    }
}

private message withdraw_checked(Booking) {
    ...
}

In case of a synchronous message the sender will receive either ok or {error, balance_too_low}. If this message has been sent asynchronously there would be no reply. You can also see that atoms and tuples should be like in Erlang, I also wouldn't change lists. Arrays are just maps with integers as keys. Public and private modifier determine if a (message) function is visible outside a module. So no more export statement is needed. The notation of higher order functions should be simplified a bit:


IoFun = $(Key, Value){io:format("~-20w = ~w~n", [Key, Value])};

maps:foreach(MyMap, IoFun);

I would do without pattern matching and guards here to be more simple. If this is needed the according functions should be defined - and well documented - and passed with a dollar sign:


private function foo({tuple, A}) { ... }
private function foo(A) when A =< 1000 { ... }
private function foo(A) when A > 1000  { ... }

lists:foreach(MyList, $foo);

These examples also show some Erlang conventions I would like to keep. Modules, processes, functions, and atoms are always named in small caps and underscores if needed. There's no mixed case, unlike in variables which are written starting with a capital letter and use mixed case.

This definition is by far not complete, but it is a cause for thought. So now I'm looking foreward to your responses.

SOA India 2008 - Travel report

2008-11-23T21:38:00.003+01:00

Since Thursday I'm back in Germany, time enough to write a little travel report. This part will concentrate on the journey itself and all my impressions. Another one about th thoughts I've made on future IT landscapes will follow in another entry soon.

The travel

Due to the strike of the Air France pilots my flight has been canceled and I've had to take another two days later. Due to the flexibility of the conference organizers it has been no problem to change the agenda. But on the other hand I've missed the chance to take part in a sightseeing tour. *sigh* Additionally on both flights we've been above India, the Ocean, Pakistan, Iran, and Iraq in the dark. I would have liked to see more. The flights themself and the service of Air France have been ok with a good entertainment and tasty meals. But, as always on long distance flights, the seats are too small. Looking at the chairs at the Business Class makes me envious. *hehe*

India

Wow, what a country! First I've been a bit sceptical, not knowing what to expect. But now I'm really keen on India. Yes, it's chaotic, the traffic with the continuous honking, the bad road conditions, those unlighted and dark trucks at night, and the incomprehensible way how it all flows. But it's also funny, so different from Germany where everything needs rules. It just works. Also the toilets are different, even if I haven't had a real traditional one. It's somewhat strange, but it's handable.

The gap between poor and rich is great and more visible than in Germany. It's embarrassing until you realize that it's normal. It seems to be ok. All the impressions when driving around are overwhelming. It starts with those foreign smells during the transfer from the airport to the town. Bengaluru itself is a fast growing town, many construction zones, but also many unfinished buildings. The music will never be my favorite one, but we've had fun dancing in the club or hearing it in the streets when driving back to the hotel. And the food, ah, yes, delicious. It's hard to describe, with all those foreign spices. It's hot, sometimes really hot. I like it.

What impressed me most are the people. They are friendly, open, and always complaisantly. And they always act service-oriented, absolutely unlike in Germany. Here we've bought some shoes on Friday. No service personnel, we've got to wait until there has been someone to take the money, and she made a face as if we're disturbing her. What a difference. So I like the Indian people and I'm looking forward to see them again during a future conference, especially Riti. She's very kindly, I really enjoyed to work together with her.

But what really confused me has been the face of the taxi driver when I drove to the airport. It has been a 2h tour and we talked a lot about the differences of our countries. Beside the chance of seeing more of Bengaluru here I really enjoyed our tour. But when I thanked him for this pleasant trip he looked at me really puzzled. It doesn't seem to be common.

The conference

Unfortunately I've not been able to visit the Monday sessions. And on Tuesday I've had three tracks - due to the interest of the audience I've repeated the session On Event-Driven Architecture - and two open network sessions. The last day had been a full day workshop by Axel. All in all the focus had been technologically, not so much on business cases. This may change in future. Alltogether S&S is satisfied with the reception of the audience.

I'm also almost satisfied with my sessions. During the first one I've been a bit nervous. It has been my first international presentation in English. But number two and three have been ok. It makes fun. Photos and perhaps videos will be available soon, as well as all the slides. My private ones can be seen at http://picasaweb.google.de/themue/SOAIndia2008.

So altogether I've enjoyed it and I would like to participate actively in more conferences in future, in India and elsewhere.

SOA India 2008 - Last day

2008-11-19T07:43:00.000+01:00

Today it's my last day in India. I'll leave at 2:15 in the night. Until then I'm following the workshops, take some photos, and maybe after lunch I'll do some sightseeing. India - at least Bangalore - is really interesting with friendly people all around. But it's also loud, chaotic, and has a horrible traffic. All together it's a great experience.

SOA India 2008 - Sessions done

2008-11-18T13:33:00.000+01:00

So, all my sessions are done. At the first one I have been a bit nervous, but the second one has been pure fun. And the unplanned repeating of the first one due to the interest also. So all in all it really has been a great experience. All other impressions I'll post later.

Stay tuned.

SOA India 2008 - Short night, warm day

2008-11-18T06:17:00.003+01:00

Tomorrow morning at 00:40 local time I arrived in Bangalore. And it lasts until 02:30 until light in hotel where switched of. The day started at short past 7, so it has been a real short night. *yawn* Currently it is almsot 11 and it's already very warm. I'll see what the day will bring.

SOA India 2008 - Je suis en Paris

2008-11-17T08:58:00.003+01:00

I've just reached my second stop: the Charles de Gaulle airport in Paris. It has been a wonderful landing, seeing the awakening town with all the lights. Even if I've been able to see the Seine I could not detect the Eiffel tower. So now I'm waiting here at the terminal 2C and wait for my flight to Bangalore. Currently it's expected on time. Thankfully the WLAN of the Emirates Lounge is open. *smile*

Stay tuned.

PS: I didn't expected so awful toilets here.

SOA India 2008 - Here I come

2008-11-17T05:46:00.000+01:00

This time it seems to be successful. I'm sitting here at the airport Bremen, checked-in, no flight canceled so far. So India, here I come.

Stay tuned.