EAI using shared database?? probably not!

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

Enterprise software inevitably involves integration. The approach of building individual monolithic applications to solve enterprise problems has evolved into an approach of dividing the problem into smaller ones, coming up with different solutions for those and finally having an integration mechanism in place which puts everything together.

This approach not only simplifies the problem at hand, but also makes evolution of software easier and reduces the "single point of failure" problem with monolithic apps.
Well...the valididy of the above statement depends on the way it is done. This approach can lead to probably bigger problems if enough care is not taken at the right times.

Talking about the integration bit, there are various approaches to choose from. And these are the most common ones. File Transfer, Shared Database, Some form of Messaging, etc. Of course, as always, each approach has its pros and cons, but largely, here are my notes on why i feel that the shared database approach seems lika a bad idea.

Multiple applications accessing the same database is quite analogous to the shared memory way of implementing concurrent access of data (Consider each application as a thread and all threads accessing a shared data structure which is the database in this case). The shared memory approach comes along with its set of patterns and best practices to be applied for a successful implementation -

• have as minimal shared data as possible
  
• synchronizing access to the shared data structure
  
• not having copies of that shared data in multiple threads
  
• do as little as possible in the synchronized block (i.e. when the lock has been acquired)
  -probably just do the actual update of the data and free the lock. No thread should hold the lock for long periods.
  

More details could be found here (Warning: PDF file)

The shared database EAI approach will also involve taking corresponding equivalent precautions, Which in a lot of cases, is tough to achieve. All applications will be talking to the same set of tables - minimizing shared data in this case means restricting use of certain tables by only specific apps and so on - which leaves us with few tables which are used as a communication medium between apps. And again this, over a period of time, tends to spill over and might end up in a situation where apps start talking to unrelated tables.

Another common pattern seen in such apps is that the database ends up being used as a queue - filled up by one app and consumed by another.

Caching data in the applications takes a serious hit as the data might get stale more often than not. Performance, being the main reason behind caching, goes for a toss too. This is corresponding to the "not having copies of that shared data in multiple threads" as mentioned above. Caching it is infact keeping a copy of the global data which might change without notice. What might end up happening is that version checks are done every now and then, or atleast on updates on all tables.

Taking a step backward, there are questions about the shared memory way of implementing threads itself.
In highly concurrent programs, languages like Erlang are preferred over others which provide a shared memory based solution. This is tending towards the messaging approach which erlang uses and hence totally avoids shared data. Again, it depends. Probably a combination of both might be the best solution in some cases as well.

Taking one more step backward, looking at this approach in an abstract sense, it can be seen that the data would not encapsulated. All applications have access to all data. It just doesn't look like the right way of doing it, unless there are enough justifications for it. It looks similar to having global variables in the code (Which is also avoided as far as possible).

Rather, consider that each application talks to its own database (which holds just enough data which only this app concerns itself with) and the different apps communicate by sending messages to each other(which is close to an SOA architecture). A common objection to this approach is: There is a lot of data to be sent across to another application. If XML is used, serialization and deserialization itself becomes an overhead etc etc.

My answer to that would probably be: The very fact that a lot of data has to be sent is probably indicating that the responsibilities of the applications should be refined. Maybe there is another application in the middle. Maybe some of the operations exist elsewhere.
The crux of the problem is in effectively identifying and conceptualising the applications (or services), just as identifying and conceptualising objects is vital for OO.

However, a few good articles about EAI approaches are here, here and here

Kickstart, and a parable

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

This will be the place where I scribble my observations and thoughts on computer science, technology, programming, and related topics. Felt the need to separate these things from my other blog, which deals with other aspects.

I came across this funny story recently. Funny it seems, but it is not far from reality. Read it here.
There can be a variety of conclusions drawn from it, I don't wanna say anything more.

Snow at Work

over 3 years ago | Karthik: Chain of Thoughts

Probably the snow had taken off for the weekend. We knew that it was round the corner because of the clouds and the wind.

And promptly enough, on monday morning, just when we were having breakfast at the Hotel, it was back to work. It snowed and snowed, and continued to snow for hours together till afternoon. And what was left back was a whole new white picture of the place.

The old taxi driver says, "The sun'll come out soon and clean all this away". High hopes. But that that was just the beginning. For it again snowed after a couple of days, only heavier this time, accumulating more work for the sun. And also for a lot of people - people who clean the snow off the roads, the traffic police who'll have to be on the lookout for accidents, and so on.

It was just beautiful! The way the snow settles down on the ..on practically everything, is a sight to see.
With Christmas in the vicinity, all the trees and buildings are wonderfully decorated, and along with the snow adding to it, its all the more impressive. The festive mood strongly strikes one who walks through the streets.

By saturday, the snow again took a break and the sun started. It took more than two days to clear all the backlog and now the streets are back to their original colours.
But I don't think its for long, though. The snow is just lurking behind somewhere and will soon be back I'm sure.

P.S.: Click on the image for actual size