I want to put a ding in the universe.

For the last month we've been working in Salzburg to build such a system, the Intelligent Project Controlling Tool needed for running large collaborative research projects like IKS. Since the design we went with can be reused for other business analytics needs, I wanted to write a bit about it.

But first, here is how our system looks like:

Where does the data come from?

There are many ways to gather business data. Often the information systems already contain the data needed. But it may also be hidden in a jungle of spreadsheets. Or maybe some data is simply not available, and has to be filled in manually.

Handling all these cases in one system is a tricky question. To solve it, we went with a two-layered strategy:

• All data used for analytics is stored as Linked Data in a CouchDB system
• NoFlo workflows are used for gathering data from the diverse sources and convert it to the format needed

In IKS's case, much of the data was available in a series of spreadsheets. With these, we built the necessary workflows for first converting the spreadsheets into XML with Apache Tika, and then extracting the information from them in a sensible subset of JSON-LD.

Because IKS is a collaborative project, information needs to be gathered from a diverse group of partner organizations. Some of them have systems that provide the needed APIs (like Basecamp, which we use), and we can just periodically import the data. But with many we decided on a simple data interchange approach: spreadsheets handled over email.

In this approach, user files a data request into the system. This gets picked up by NoFlo, which sends an email with the appropriate spreadsheet template to the partner. Then it starts waiting for a reply. When a reply arrives, it extracts the data from the attached spreadsheet and imports it to the system.

Our NoFlo processes are mostly initiated by the CouchDB change notification API. We keep them running persistently using forever Node, so whenever some operation needs to be run it happens nearly immediately.

Ensuring data consistency

With any automation, and especially with the email-based data interchange, things can go wrong. Because of this we tag all data that we receive with its origin, whether it was some automated operation or an imported spreadsheet. These origins are called execution documents. Users can browse all completed workflow executions and see what data came in from them. These can then be either accepted or rejected.

This way if some partner accidentally sends faulty data, or something else breaks, the incorrect information received can be easily removed. CouchDB's versioning capabilities help here.

Analyzing the data

CouchDB is built on top of the concept of map/reduce. Here you can modify and combine the data in lots of different ways using simple JavaScript functions. In our case we elected to write all our CouchDB code in CoffeeScript for simplicity. For example, here is the reduce function in CoffeeScript that counts totals of time planned, time used, and time left per task or partner in a project:

(keys, values, rereduce) ->
    roundNumber = (rnum, rlength) ->
        Math.round(parseFloat(rnum) * Math.pow(10, rlength)) / Math.pow(10, rlength)
    data =
        planned: 0.0
        spent: 0.0
        left: 0.0

    if rereduce
        for reducedData in values
            data.planned += reducedData.planned
            data.spent += reducedData.spent
        data.left = data.planned - data.spent
        return data

    for doc in values
        if doc['@type'] is 'effortallocation'
            data.planned += roundNumber doc.value, 1
        if doc['@type'] is 'effort'
            data.spent += roundNumber doc.value, 1
    data.left = roundNumber data.planned - data.spent, 1
    return data

If you figure out a new way to look at the data you have, simply write the needed map and reduce functions and save them into the database. CouchDB will then run them against existing data and produce numbers.

Data visualizations

Numbers are good, but to really see the information buried in them you need some visualizations. For this we decided to follow the CouchApp idea where the user interface code is stored in the database together with the data itself. This way no application servers are needed, and you can take the whole system with you just by replicating the database. Think of the possibility of doing some analysis on your company while flying to a meeting!

The visuals are in our case provided by JavaScript InfoVis Toolkit, a nice, MIT-licensed interactive graph library.

CouchDB views handle the number crunching, then CouchDB list functions process the numbers into the format needed for visualization. This leaves only a minimal amount of work for the client side.

For consistency our application has been built with CoffeeApp, so all the database and user interface code is in CoffeeScript.

In a nutshell

Any business analytics system dealing with moderate amounts of data can be built following this approach.

• Apache CouchDB is the central data store
• All data is stored as JSON-LD entities
• NoFlo handles all data imports
• Analytics based on the data are done with CouchDB map/reduce
• Visualization happens with a CouchApp using JavaScript InfoVis Toolkit

This way you have a business analytics environment that is easy to extend with more data when it becomes available. New analysis can be done by writing reasonably simple map/reduce functions, and CouchDB's replication capabilities allow you to take the system and data with you.

Using JSON-LD for the data storage makes a lot of sense, as this way the relations between different pieces of information are easy to handle. And using URIs for data identifiers means you can easily mash up information coming from different sources together.

The two-layered approach of using NoFlo for data imports, and CouchDB for analysis also allows for clean separation of concerns. In our case, I did the workflow part of things, and Szaby built the visualizations.

Twenty-two percent — more than one in five — of America's kids live in poverty, according to 2010 stats from the Bureau.

Grouping all Minnesota children together in 2009, the poverty rate was 14 percent, but among African-American kids it was 47 percent, among Hispanic children 32 percent and among Asians 22 percent. White, non-Hispanic youngsters figured at 8 percent.

Dennis Cauchon and Barbara Hansen report on the drop in income and increase in poverty.

Michael Stern Hart

At a time when "Net Neutrality" and digital access is a non-given, one is reminded of Hart's use of mainframe time for free (in *both* meanings of the word) information distribution.

Hart related that after his account was created on July 4, 1971, he had been trying to think of what to do with it and had seized upon a copy of the United States Declaration of Independence, which he had been given at a grocery store on his way home from watching fireworks that evening. He typed the text into a teletype machine but was unable to transmit it via e-mail. Thus, to avoid "crashing the system", it had to be downloaded individually.

- Wikipedia

Hart's keen intelligence and tenacity are described in a letter of recommendation by his Assistant Dean at the University of Illinois posted on The Project Gutenberg Literary Archive Foundation website:

I feel strongly that the keenness of his intellect and the quickness of his mind, his personal stamina and his fearless determination to pursue avenues of inquiry which frighten the less vigorous minds are worth of serious consideration for a position in which innovation and compelling imagination are essential.

Today Project Gutenberg, which Hart founded, offers over 36,000 free ebooks to download and use on a variety of digital devices. Over 100,000 books are distributed by the organization's affiliates.

You want to see my stressed face?

The second edition of Fortnight, the new multimedia online journal assembled quarterly in New York, features 14 contributors from 14 disciplines. That's 56 young people annually, their actions today and ideas of tomorrow.

Masthead lists: Adam Whitney Nichols and Samantha Hinds as Co-Founders, Ian Lewis Campbell, Managing Editor and Patti Smith, Patron Saint. Fortnight is a Solo Foundation project.

The bridge crossing the lagoon in Boston's Public Gardens was erected in 1867.

Even in the ten years of history of our company this is quite a significant move - it allows us to combine Nemein's traditional expertise on content management with Infigo's mobile offerings. As smartphones and tablets are becoming popular, more and more services we build will have a mobile element, which is now easier with lots of in-house expertise.

This also means more focus on the interplay between the Midgard content repository, NoFlo workflows, Node.js and Symfony web services, and mobile applications built in Qt.

Petri Rajahalme (with me in the photo) will be the CEO of the merged company, and I will focus on leading the R&D efforts.

After the recent San Francisco MeeGo Conference there has been surprisingly much negative reporting about MeeGo, mostly centered at Nokia's MeeGo story. While Nokia's strategy changes are unfortunate, much of the reporting around it appears to come from misunderstanding what MeeGo is about.

Many see MeeGo just as Android without Java, but it is much more, as I'll explain here.

Industrial Linux

MeeGo is much more than just handsets or tablets. It is an attempt at creating a standardized industrial Linux distribution that can be used anywhere from in-vehicle infotainment devices to TVs to, indeed, handsets.

It is a true open and collaborative environment, managed by Linux Foundation. The governance model is there to ensure that MeeGo stays a vendor-neutral platform that anybody can build their products on top.

Many device segments have very long development, and especially usage times. For this MeeGo has a predictable release schedule of a major release every six months, and a roadmap kept by the Technical Steering Group.

If MeeGo succeeds in this, you will be using it in your TV, in your car stereo, and at the back of an airline seat. But in most of these situations you won't be able to know that it is MeeGo. It is simply there to make building products faster and cheaper for the manufacturer.

Openness

As I argued in my earlier piece Open Source? Free Software? What we need is Open Projects, being an open platform is much more than just the licensing terms of the code. There needs to be transparency into the development process, a clear procedure on how to participate and much more. And of course licensing has to be such that the participants can actually use the results in whatever they're doing.

For this, most of MeeGo is licensed under permissive terms, like the GNU LGPL and BSD-style licenses.

But indeed, the other aspects of openness are more important. With MeeGo you can see every commit happening on Gitorious, and you can see the bugs and features being worked out in a public Bugzilla.

MeeGo as a project is still quite young, and many participants are still learning how to work in the open. This has lead to some issues in project transparency. But hopefully those are now getting resolved.

User Experience

MeeGo allows anyone to build their own user experience on top of the platform. Actually, this is expected of any serious manufacturer. Sure, there are some reference UXs available, including Tablet, Handset and Netbook, but none of these are quite product-ready, and are not necessarily even intended to be.

Because of this it is quite funny to see reviews of the reference UXs. They're not the ones most devices will run, though obviously some manufacturers or community members are going to use them anyway. A full MeeGo product will look and feel like something completely different.

This is not like Android manufacturers adding their own skins. With MeeGo anybody has the full freedom to build a complete user experience that suits their device, branding and other goals. The whole platform has been built to allow this sort of differentiation, without a risk of fragmenting the ecosystem. I'll explain the fragmentation question soon.

Actually, the freedom of defining your own user interface is big enough that both Android and WebOS could theoretically be rebased on top of MeeGo to be just different MeeGo UXs. Obviously they would need to allow running MeeGo-compliant Qt applications in addition to ones written for them directly, but that is minor detail. WebOS already ships Qt, so it isn't even that far from this. Similarly, KDE or GNOME could run as MeeGo UXs.

Compliance

At the core of MeeGo there is a set of compliance rules. Being Open Source, anybody can take MeeGo, modify it, and run it on their devices. But only if their implementation passes MeeGo compliance it can be called MeeGo.

Device Compliance is a set of rules that ensures any MeeGo-compliant software can run on a particular device. Application Compliance similarly ensures an app can be installed and run on any MeeGo-compliant device.

Both of these sets of compliance rules have automated tests that anybody can run. So, between non-compliant MeeGo-related software there may be fragmentation, but anything branded MeeGo (and therefore compliant) must be fully compatible.

App Stores and business models

MeeGo is an open source project, not a company. This means it comes without strings attached, compliance rules aside. There are no limitations on the business model of a MeeGo device manufacturer, no mandatory online services or app stores to enable, and no royalty payments.

With this, each vendor can decide what they want to enable their users to do with the device. An embedded device might have no concept of installable applications, a tablet might come with the vendor's own app store.

For those who do not want to go through trouble of building their own developer ecosystems and app stores, there are some generic solutions available in the MeeGo sphere:

Intel's AppUp is a "white label" app store. This means that a device manufacturer, or even retailer or operator can get an instance of AppUp with their own branding and a revenue sharing deal with Intel. Developers submit software only once and it will be available on all the different branded AppUps.

On the more open side, there is also the upcoming MeeGo Community Apps, a fully community driven "store" of free software written for MeeGo. It comes with its own, OCS-compatible client application, a web frontend, and clear set of crowdsourced app quality assurance processes. The similarly handled Maemo Downloads has served over 80 million downloads for the Nokia N900, so the user and developer interest is clearly there.

The future of MeeGo

At this early stage of the project it is hard to make predictions, but there are many things MeeGo gets right. I think it has a bright future ahead of it, especially in more specialized devices. There the shared infrastructure and clear development schedule give manufacturers substantial advantages in both development time and cost.

Product development times in the embedded sector are quite long, and it may well take years before we'll see MeeGo in a airline multimedia system. But if the project shows the necessary durability and longevity, this will eventually happen. Now many of those systems run on customized Linux distributions that their manufacturers have to spend quite a bit of money to maintain. MeeGo removes that problem, and allows easier collaboration through the compliance rules.

As for consumer devices like tablets and handsets, that area mostly requires there to be a vendor that wants to properly differentiate itself from the grey masses of the Android ecosystem. MeeGo provides all the necessary tools on both systems side and user interface development to make that happen.

Currently there are many different ideas floating around on how to build user experiences on connected devices. There is the "wall of apps" approach of iPhone, there are the fully cloud-connected WebOS and Android approaches, and now Microsoft is also starting to enter the game with their own ideas.

I don't think the "post-PC" world is yet complete. What MeeGo gives is a fast way to build products differentiating from that crowd. It just needs companies who are willing to go for it.

The next couple of years will be quite interesting.

Tai the elephant

Watching the last third of the movie was plagued by what appeared to be a loose or failing projector bulb, flickering as the entire screen went black in annoying flashes. Closing one's eyes solved the visual annoyance, but it defeated the filmmakers' purpose of spending tens of millions to make even a mediocre movie. A couple of patrons went to notify the absent management.

Technical problems happen, but we did notice how dim the projection was even without the flashes of black. Ty Burr reveals why.

Not so apparent if you use the wrong lens

Some theater owners, the same ones who complain about Video-On-Demand and the shrinking Theater-to-DVD Window, are too cheap to change 3-D lenses on Sony digital projectors (which they got for free) when showing a film that is not 3-D.

The old "open vs. proprietary" debate is over and open won. As IT infrastructure moves to the cloud, openness is not just a priority for source code but for standards and APIs as well. Almost every vendor in the IT market now wants to position its products as "open." Vendors that don't have an open source product instead emphasize having a product that uses "open standards" or has an "open API."

"Openwashing" is a term derived from "greenwashing" to refer to dubious vendor claims about openness. Openwashing brings the old "open vs. proprietary" debate back into play - not as "which one is better" but as "which one is which?"

Especially Google seems to be doing this quite a bit. If you want to be open, work in the open. This is the only way to ensure acceptance and sustainability for your code.

Facebook may have a huge installed base, but it's dead to me. I can't get there. The platform vendor is too active. Same with Twitter, same with Apple. Give me a void, something I can develop for, where I can follow the idea where ever it leads. Maybe there are only a few thousand users. Maybe only a few million. Hey, you can't be friends with everyone.

- Dave Winer on Twitter's new developer roadmap

The best laid plans

The initial plans were made at a Starbucks on New York's JFK airport while waiting for a flight to Atlanta, but their realisation had to wait until I finished my military service on the latter half of 2000. When I got rid of the bazookas and uniforms, we registered the company, wrote some business plans and started looking for seed money to get our business started. We were quite young then, and it was interesting to run around Helsinki talking to investors.

How did these plans look like? Our initial idea was to get into the fashionable SaaS (or ASP, as it was known then) business by building collaboration tools on top of Midgard. The first product was a document store intended for the construction industry. With this system all plans and other documents related to a building project could be easily stored and accessed. This is how we described ourselves:

Nemein Solutions is the leading provider of Open Source Midgard software for mobile collaboration and information management.

But as plans go, this had to soon change due to the IT bubble being burst. To quote von Moltke:

No plan of operations extends with certainty beyond the first encounter with the enemy's main strength

In spring 2001 IT bubble burst, and we suddenly found ourselves sitting in the office with all our projects being abruptly frozen. Around the same time our seed investor got embroiled in some large-scale customs lawsuit, and so not much help was to be expected from them. This meant we couldn't continue with our original plans, and instead had to start generating cash flow, quickly. Luckily Midgard was (and is!) a quite capable web framework, and so we had the option of going into the CMS business.

Nadmin Studio

Midgard's user interfaces back then were not very appealing, and so our first task was to go shopping for the CMS UI. There were two good options available: Nadmin Studio from Hong Kong Linux Center, a web based CMS and small business networking tool running on top of Midgard, and a Windows-based Midgard editing tool from DataFlow. As we were much more of a Linux shop, we went with Nadmin. It was quite a cool system, a customized Red Hat Linux install that set up not only Midgard and the web user interface, but also things like LDAP and IMAP servers talking directly with the Midgard database. And it had a quite nice WYSIWYG editor for people writing content on the web pages. We quickly became their reseller for Finland. Yes, back then you could get Midgard in a box (and even CD):

Having settled the tool question the next issue was finding clients. We took a list of hundred largest companies in Finland and basically called each of them, proposing a demo. We also approached several "new media companies" in order to see if they wanted a technical partner. Around these times our CEO Petri Kuusela also figured that we'd be a lot more convincing consultants in sweaters instead of Hugo Boss suits, and so the look of the company changed.

Through these efforts we were able to get some of our first and longest-term customers, including this one:

HELSINKI, Jun 12th 2001 -- Nemein Solutions helps Everscreen Mediateam, a Finnish multimedia company implement the Nemein.net Content Manager product to power Motiva's web services. Everscreen's and Nemein's cooperation provides Motiva with up-to-date and easy to use web sites.

Around this time we moved from the small four-desk office in central Helsinki to a much bigger place in Haukilahti, Espoo. My time was mostly spent motorcycling from one demo to another, as our two sales guys kept me so busy that on most days I didn't have time for a lunch break, and much less for actually writing code. The cash flow generated there helped to keep things running, but as usual, possibilities for product development suffered. This is called the Consulting Trap:

...Once the consultancy money rolls in, it is hard to give up. Like an addiction.

I spent years thinking just six more months, then I'm going to quit and work on my µISV project.

Nemein.Net

Consulting isn't such a bad business to be in. As long as the things you do produce value for customers it can be lucrative and interesting. But still the idea of having actual products was kept alive, and a bit later we built Nemein.Net, a project management tool for consulting companies. We changed the business model a bit, instead of providing hosted services we leased some industrial-grade servers to our clients with the software pre-installed. A cluster of engineering companies bought that, and as far as I know some of them still run it. Datex-Ohmeda was another customer, but they were later bought by General Electic.

"For our project work it is very important to reduce management overhead and enable real-time tracking of project status. The Nemein.Net Projects suite provides a good match for these criteria," says Bror-Eric Granfelt, R&D Manager at Datex-Ohmeda.

Free Software company

In 2004 we open sourced the Nemein.Net suite, now renamed to OpenPSA. This was done as part of the 5th anniversary celebrations of the Midgard project. By this time Nadmin Studio had also been GPLd and renamed to Aegir CMS. So suddenly we were a pure Free Software company. We quickly started adopting MidCOM, the emerging MVC framework for Midgard and PHP. MidCOM was produced initially by the German ISP Linksystem Muenchen, but very soon Nemein was the primary contributor.

The company structure changed, and we decided that instead of having a traditional office with desktop computers, it'd be better to be more location independent and work where our customers were. So we got a small office from the Innopoli business park mostly to facilitate Rambo, and the rest of our people were moving around. Once a week we had a coordination lunch meeting in Restaurant Mount Everest to keep the group spirit going. Some of that tradition has stayed.

Around this time we also started the switch on our workstations from HP's Linux laptops to MacBooks. This wasn't really a conscious strategy, but instead mandated by my laptop breaking a day or two before a training trip to South Africa. Back then Linux on laptops was still a quite cumbersome setup, and I needed a Unix machine where our software would run, quick. Later on I've returned to running Linux on my own machines, but most of the company still works on OS X.

The Finnish Centre for Open Source Solutions (COSS) was formed in 2003, and we soon joined up. A forming network of free software companies in Finland was good for both publicity and getting new projects. OpenPSA gained a boost there:

Collaboration with Nemein went well. Right from the beginning they were able to state their views clearly and backed by facts. We immediately understood how OpenPSA works, what customizations would be needed, and how much they would cost. Unfortunately we cannot say the same of all other solution providers, says doctor Ville Ojanen.

Dreams of networked business

Another interesting opportunity that came from the COSS network was the EU-funded Digital Business Ecosystems project. The project fit quite well in my view of the need for enabling cooperation between small companies in Europe, this time through having business systems talk to each other over a peer-to-peer network.

To realise this dream we connected our OpenPSA system into the ecosystem, enabling companies to fluidly share tasks, workflows and hour reports over the network. Unfortunately not much came out of that. A bit later the maintenance of the OpenPSA project was switched over to Content Control from Germany.

A later iteration of similar ideas was Ajatus, an experimental project to build a "personal CRM".

"Companies that don't realize their markets are now networked person-to-person, getting smarter as a result and deeply joined in conversation are missing their best opportunity." - The Cluetrain Manifesto, these 18.

Remember a time when you needed to share a document with a business partner, colleague or a customer? The CRM should make this easy without requiring complex IT integration setups or the disconnectivity of emailing files.

Getting into position

Several Nemein people have been active motorcycle travelers. As all our projects were more or less visible on the web, this brough the question of location sharing into the picture. For the Death Monkey project in 2006 we built a set of location-aware features that enabled us to visualize the location of each participant on a map, and easily calculate distances to Gibraltar.

At that time using maps on the web was also growing, and so we stepped into the emerging business of neogeography. Over the years we've evangelized the usage of location information on Linux desktops, built weather-aware clothes catalogues, facilitated publishing open data of campus maps and made it easier to catch a tram. This is still one of the areas online that I find most interesting.

Growth and mobility

Over the years Nemein's business has been growing at a steady pace. Now we have a nice small office in the Hietalahti area of Helsinki, and serve quite a bunch of interesting, large Finnish customers in the CMS space. A major milestone for the company was achieving AAA credit rating back in 2007:

Ignited by Apple's iPhone launch, the mobile ecosystem has been a very interesting area to operate in. To be part of it, we built the community infrastructure for Maemo, Nokia's emerging mobile Linux platform, and also got involved in the MeeGo project. But now in the age of burning platforms the future of that business is in question.

The Midgard way

The Midgard content repository and web framework have been a constant core part of our business for the whole history of the company. Everything we've built has been running on top of it. Has this been a wise choice? In the course of ten years, the web landscape has changed quite a bit. While Midgard itself has stayed current through constant development and refinement, hundreds and hundreds of competing systems have risen up, some of them becoming very popular compared to us. And yet we have stayed the course.

Midgard, especially in the latest iterations, is an excellent tool for running information-rich systems. It has a very nice user interface and an elegant web development framework. These are tools that I feel have lots of possibilities still ahead of them. Some of the design decisions done in the early days of the project, like integrated support for multi-site hosting, and for multilingual content, are things that now power some of our most important customer deployments

But at the same time I've learned that especially for smaller open source projects like us, the monolithic "all or nothing" approach is not very healthy. Frameworks keep us apart, while libraries allow us to share our code and experiences. This is resulting to collaboration with other projects on many levels, from a shared PHP ecosystem managed through the Apache Software Foundation, to common tools for decoupling the Content Management experience. Linked Data also plays a large role here.

To wrap it up

Ten years as an entrepreneur is a long path. Financially it may not have been as rewarding as we initially thought it would be, but experience-wise it has been astonishing. I've been part of building many challenging business-critical systems, learnt a lot of things, and given talks in dozens of conferences all around the world. It is hard to see as varied and interesting possibilities in regular employment.

Thanks to the whole current Nemein team, and the people who've been here before for all the awesome work done over these years. You rock!

Ladies and Gentlemen, we bring you a brief musical interlude.

Happy New Year!

American Consumers do their part.

What we think is most powerful is not one modality -- it’s how multiplatform you can become. Consumers want choices.

Must be a bad hair day for Skype CEO Tony Bates whose call for Premium Customers went out on the wire today.

Companies like Skype have a tremendous amount of opportunities. You have to focus on the things that matter.

Oh well, back to Monster Mayhem.

The answer is sustainability. IKS is an EU-funded project which will eventually end. Proper project governance handled together with the Apache Software Foundation can help the software to survive and thrive for long after that.

Sustainability is something that is critical for all libraries and infrastructure software. If you want adoption, you need to ensure potential users and developers that the software will continue to be around. The way to accomplish that is to have a clear, open governance model. It is time to stop throwing code over the wall.