Erlang adopters on the rise

over 2 years ago | Karthik Ramachandra : Concrete Abstractions

Stories about new adopters of Erlang are going up day by day. Here are a few:

• Amazon SimpleDB (here)
• FaceBook's instant messaging feature(here)
• Engineyard's new cloud computing platform - vertebra (here)
• IMDB (here and here)
• Google gears uses erlang style concurrency(here)
• Scalaris - A Transactional Scalable Distributed Data Store (pdf: here)
• Apache couchDB (here)
• ...
  

Feel free to add to this list...

The Erlang Exchange 2008 - a report

over 2 years ago | Karthik Ramachandra : Concrete Abstractions

The Erlang exchange was the first international conference on erlang aimed at all erlang enthusiasts, from architects to newbies. It was also targeted at businesses, who got more visibility about erlang and its community.(about 30% of the participants were from the financial sector)

About 150 people participated in the two day conference which was held at the Old Sessions House, London. There were more than 30 speakers from various backgrounds, who came to share their experiences. People got to interact with the likes of Joe Armstrong, Klacke Wikström, Steve Vinoski, in an informal environment.

Day 1 : 26th June 2008

The Keynote was delivered by Joe Armstrong, who spoke about erlang, its potential and practicality and gave a kind of an introductory context to the conference as a whole.
This was followed by talks in two tracks, the tools and gadgets track, which included topics like Erlang DTrace, Erlang and Robotics; and the startup track, with talks on Erlang/OTP vs. Google App engine etc. Find the full schedule here.
The programme was planned so that people got to socialize with each other during coffee/tea /lunch breaks and there were always lots of interesting discussions happening.


The post lunch tracks were about web applications and financial applications in erlang. There were also talks about erlang build and packaging systems(erlware), and Wrangler ( the erlang refactoring tool).

After 6 PM, there was an informal set of sessions by the representatives of various UserGroups in London, about their communities and activities. Representatives from LRug(Ruby), Haskell User Group, Groovy User Group and a couple of other groups were present. This continued on to the Erlang Exchange party.



It was a very fulfilling day, a fun day, and a tiring one too, since there were loads of new concepts and ideas floating around all through the day.


Day 2 : 27th June 2008

Day two started with the keynote by Steve Vinoski who spoke about Enterprise integration. He traced the evolution of the enterprise industry and approaches to enterprise integration, while pointing out interesting observations on how technologies can end up creating more problems than solving it if "developer convenience" is prioritised over "correctness" of the solution.
This was followed by an interesting talk by Klacke Wikström, about erlang and tail-f systems.
After this, there were three parallel tracks for the rest of the day, with a few tutorial sessions.

The Enterprise Integration track included talks about RabbitMQ, EJabberD, JInterface, CouchDB, etc.

There was also a very interesting session by Alexander Reinefeld on "Building a transactional distributed data store with Erlang". Refer to the paper here. They have some amazing results on experiments conducted by hosting the entire Wikipedia on a p2p infrastructure, completely built with erlang which scales better with fractional hardware! Very very interesting!


The Testing Track had talks about eunit for unit testing, using erlang for Performance testing of web applications, tutorials on QuickCheck and Tsung. The other tutorials were about buiding web applications in erlang, using Faxien and Sinan, and about robotics.

The conference concluded with a panel discussion on Erlang enterprise integration. Most people agreed with Joe's opinion that erlang needs a killer app to prove to the enterprise before it can be widely accepted. And the lack of libraries for lots of small things was also another reason which would put people off erlang.

One thing that everyone in the room was convinced about was that erlang is going to definitely grow big in the coming future. And the erlang community, though quite small as of now, is quite mature and pragmatic in saying "The right tool for the right Job" - Using erlang gives tremendous results, only when used for the right things. And this pragmatism, I feel, is a very positive way to look at solving the problems in front of us.

Overall, it was a very successful event, thanks to the organizers (skillsmatter and erlang training and consulting) , sponsors, speakers and participants. The whole event has been videotaped and hopefully would be available soon.

Ubuntu Webcam story

over 2 years ago | Karthik Ramachandra : Concrete Abstractions

There was this webcam... (now you can guess the story ;-)

It refused to wake up on windows vista, it indeed slept well on xp too,
..(now comes the boring part) -
It just started playing on ubuntu. Fullstop. End of story.

All i had to do was this:

mplayer tv:// -tv driver=v4l:width=640:height=480:device=/dev/video0

Way to go!

Migrate with care!

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

Its the second time now! We found another production issue in our rails app which was because we screwed up our migrations :(
If only we had taken enough more care while writing our migrations....

Indeed, migrations are very powerful ..but needless to say, quite dangerous too if not used the right way.

Recursive or iterative?

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

"In contrasting iteration and recursion, we must be careful not to confuse the notion of a recursive process with the notion of a recursive procedure. When we describe a procedure as recursive, we are referring to the syntactic fact that the procedure definition refers (either directly or indirectly) to the procedure itself. But when we describe a process as following a pattern that is, say, linearly recursive, we are speaking about how the process evolves, not about the syntax of how a procedure is written."

Recursive Procedure for the recursive process:

(define (factorial n)
(if (= n 1)
    1
    (* n (factorial (- n 1)))))

Recursive Procedure for the iterative process:

define (factorial n)
(fact-iter 1 1 n))

(define (fact-iter product counter max-count)
(if (> counter max-count)
    product
    (fact-iter (* counter product)
               (+ counter 1)
               max-count)))

EAI using shared database?? probably not!

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

Enterprise software inevitably involves integration. The approach of building individual monolithic applications to solve enterprise problems has evolved into an approach of dividing the problem into smaller ones, coming up with different solutions for those and finally having an integration mechanism in place which puts everything together.

This approach not only simplifies the problem at hand, but also makes evolution of software easier and reduces the "single point of failure" problem with monolithic apps.
Well...the valididy of the above statement depends on the way it is done. This approach can lead to probably bigger problems if enough care is not taken at the right times.

Talking about the integration bit, there are various approaches to choose from. And these are the most common ones. File Transfer, Shared Database, Some form of Messaging, etc. Of course, as always, each approach has its pros and cons, but largely, here are my notes on why i feel that the shared database approach seems lika a bad idea.

Multiple applications accessing the same database is quite analogous to the shared memory way of implementing concurrent access of data (Consider each application as a thread and all threads accessing a shared data structure which is the database in this case). The shared memory approach comes along with its set of patterns and best practices to be applied for a successful implementation -

• have as minimal shared data as possible
  
• synchronizing access to the shared data structure
  
• not having copies of that shared data in multiple threads
  
• do as little as possible in the synchronized block (i.e. when the lock has been acquired)
  -probably just do the actual update of the data and free the lock. No thread should hold the lock for long periods.
  

More details could be found here (Warning: PDF file)

The shared database EAI approach will also involve taking corresponding equivalent precautions, Which in a lot of cases, is tough to achieve. All applications will be talking to the same set of tables - minimizing shared data in this case means restricting use of certain tables by only specific apps and so on - which leaves us with few tables which are used as a communication medium between apps. And again this, over a period of time, tends to spill over and might end up in a situation where apps start talking to unrelated tables.

Another common pattern seen in such apps is that the database ends up being used as a queue - filled up by one app and consumed by another.

Caching data in the applications takes a serious hit as the data might get stale more often than not. Performance, being the main reason behind caching, goes for a toss too. This is corresponding to the "not having copies of that shared data in multiple threads" as mentioned above. Caching it is infact keeping a copy of the global data which might change without notice. What might end up happening is that version checks are done every now and then, or atleast on updates on all tables.

Taking a step backward, there are questions about the shared memory way of implementing threads itself.
In highly concurrent programs, languages like Erlang are preferred over others which provide a shared memory based solution. This is tending towards the messaging approach which erlang uses and hence totally avoids shared data. Again, it depends. Probably a combination of both might be the best solution in some cases as well.

Taking one more step backward, looking at this approach in an abstract sense, it can be seen that the data would not encapsulated. All applications have access to all data. It just doesn't look like the right way of doing it, unless there are enough justifications for it. It looks similar to having global variables in the code (Which is also avoided as far as possible).

Rather, consider that each application talks to its own database (which holds just enough data which only this app concerns itself with) and the different apps communicate by sending messages to each other(which is close to an SOA architecture). A common objection to this approach is: There is a lot of data to be sent across to another application. If XML is used, serialization and deserialization itself becomes an overhead etc etc.

My answer to that would probably be: The very fact that a lot of data has to be sent is probably indicating that the responsibilities of the applications should be refined. Maybe there is another application in the middle. Maybe some of the operations exist elsewhere.
The crux of the problem is in effectively identifying and conceptualising the applications (or services), just as identifying and conceptualising objects is vital for OO.

However, a few good articles about EAI approaches are here, here and here

Kickstart, and a parable

over 3 years ago | Karthik Ramachandra : Concrete Abstractions

This will be the place where I scribble my observations and thoughts on computer science, technology, programming, and related topics. Felt the need to separate these things from my other blog, which deals with other aspects.

I came across this funny story recently. Funny it seems, but it is not far from reality. Read it here.
There can be a variety of conclusions drawn from it, I don't wanna say anything more.