Are Master Data Management and Hadoop a Good Match?

Master Data is the critical electronic information about the company we cannot afford to lose. Accordingly, we should sanitise it, look after it, and store it safely in several separate places that are independent of each other. The advent of Big Data introduced the current era of huge repositories ?in the clouds?. They are not, of course but at least they are remote. This short article includes a discussion about Hadoop, and whether this is a good platform to back up your Master Data.

About Hadoop

Hadoop is an open-source Apache software framework built on the assumption that hardware failure is so common that backups are unavoidable. It comprises a storage area and a management part that distributes the data to smaller nodes where it processes faster and more efficiently. Prominent users include Yahoo! and Facebook. In fact more than half Fortune 50 companies were using Hadoop in 2013.

Hadoop – initially launched in December 2011 ? has survived its baptism of fire and became a respected, reliable option. But is this something the average business owner can tackle on their own? Bear in mind that open source software generally comes with little implementation support from the vendor.

The Hadoop Strong Suite

  • Free to download, use and contribute to
  • Everything you need ?in the box? to get started
  • Distributed across multiple fire-walled computers
  • Fast processing of data held in efficient cluster nodes
  • Massive scaleable storage you are unlikely to run out of

Practical Constraints

There is more to Hadoop than writing to WordPress. The most straightforward solutions are uploading using Java commands, obtaining an interface mechanism, or using third party vendor connectors such as ACCESS or SAS. The system does not replace the need for IT support, although it is cheap and exceptionally powerful.

The Not-Free Safer Option

Smaller companies without in-depth in-house support are wise to engage with a technical intermediary. There are companies providing commercial implementations followed by support. Microsoft, Amazon and Google among others all have commercial versions in their catalogues, and support teams at the end of the line.

Check our similar posts

How DevOps oils the Value Chain

DevOps ? a clipped compound of development and operations – is a way of working whereby software developers are in a team with project beneficiaries. A client centred approach extends the project plan to include the life cycle of the product or service, for which the software is developed.

We can then no longer speak of a software project for say Joe?s Accounting App. The software has no intrinsic value of its own. It follows that the software engineers are building an accounting app product. This is a small, crucially important distinction, because they are no longer in a silo with different business interests.

To take the analogy further, the developers are no longer contractors possibly trying to stretch out the process. They are members of Joe?s accounting company, and they are just as keen to get to market fast as Joe is to start earning income. DevOps uses this synergy to achieve the overarching business goal.

A Brief Introduction to OpsDev

You can skip this section if you already read this article. If not then you need to know that DevOps is a culture, not a working method. The three ?members? are the software developers, the beneficiaries, and a quality control mechanism. The developers break their task into smaller chunks instead of releasing the code to quality control as a single batch. As a result, the review process happens contiguously along these simplified lines.

Code QC Test ? ? ?
? Code QC Test ? ?
? ? Code QC Test ?
? ? ? Code QC Test
Colour Key Developers Quality Control Beneficiary

This is a marked improvement over the previously cumbersome method below.

Write the Code ? Test the Code ? Use the Code
? Evaluate, Schedule for Next Review ?

Working quickly and releasing smaller amounts of code means the OpsDev team learns quickly from mistakes, and should come to product release ahead of any competitor using the older, more linear method. The shared method of working releases huge resources in terms of user experience and in-line QC practices. Instead of being in a silo working on its own, development finds it has a richer brief and more support from being ?on the same side of the organisation?.

The Key Role that Application Program Interfaces Play

Application Program Interfaces, or API?s for short, are building blocks for software applications. Using proprietary software-bridges speeds this process up. A good example would be the PayPal applications that we find on so many websites today. API?s are not just for commercial sites, and they can reduce costs and improve efficiency considerably.

The following diagram courtesy of TIBCO illustrates how second-party applications integrate with PayPal architecture via an API fa?ade.

Working quickly and releasing smaller amounts of code means the OpsDev team learns quickly from mistakes, and should come to product release ahead of any competitor using the older, more linear method. The shared method of working releases huge resources in terms of user experience and in-line QC practices. Instead of being in a silo working on its own, development finds it has a richer brief and more support from being ?on the same side of the organisation?.

imgd2.jpg

The DevOps Revolution Continues ?

We close with some important insights from an interview with Jim Stoneham. He was general manager of the Yahoo Communities business unit, at the time Flickr became a part. ?Flickr was a codebase,? Jim recalls, ?that evolved to operate at high scale over 7 years – and continuing to scale while adding and refining features was no small challenge. During this transition, it was a huge advantage that there was such an integrated dev and ops team?

The ?maturity model? as engineers refer to DevOps status currently, enables developers to learn faster, and deploy upgrades ahead of their competitors. This means the client reaches and exceeds break-even sooner. DevOps lubricates the value chain so companies add value to a product faster. One reason it worked so well with Flickr, was the immense trust between Dev and Ops, and that is a lesson we should learn.

?We transformed from a team of employees to a team of owners. When you move at that speed, and are looking at the numbers and the results daily, your investment level radically changes. This just can’t happen in teams that release quarterly, and it’s difficult even with monthly cycles.? (Jim Stoneham)

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
Could Kanban Be?Best for Knowledge Workers?

Knowledge Workers include academics, accountants, architects, doctors, engineers, lawyers, software engineers, scientists and anybody else whose job it is to think for a living. They are usually independent-minded people who do not appreciate project managers dishing out detailed orders. Kanban project management resolves this by letting them choose the next task themselves.

The word ?Kanban? comes from a Japanese word meaning ?billboard? or ?signboard?. Before going into more detail how this works let’s first examine how Japanese beliefs of collaboration, communication, courage, focus on value, respect for people and a holistic approach to change fit into the picture.

The Four Spokes Leading to the Kanban Hub

  1. Visualise the Workflow ?You cannot improve what you cannot see. The first step involves team members reducing a project to individual stages and posting these on a noticeboard.
  2. Create Batches ? These stages are further reduced to individual tasks or batches that are achievable within a working day or shift. More is achievable when we do not have to pick up where we left off the previous day.
  3. Choose a Leader the Team Respects – Without leadership, a group of people produces chaotic results. To replace this with significant value they need a leader, and especially a leader they can willingly follow.
  4. Learn and Improve Constantly ? Kaizen or continuous improvement underpins the Japanese business model, and respects that achievement is a step along the road, and not fulfilment.

The Kanban Method in Practice

Every Kanban project begins with an existing process the participants accept will benefit from continuous change. These adjustments should be incremental, not radical step-changes to avoid disrupting the stakeholders and the process. The focus is on where the greatest benefits are possible.

Anybody in the team is free to pull any batch from the queue and work on it in the spirit of collaboration and cooperation. That they do so, should not make any waves in a culture of respect for people and a holistic approach to working together. All it needs is the courage to step out of line and dream what is possible.

The Kanban Project Method ? Conclusions and Thoughts

Every engine needs some sort of fuel to make it go. The Kanban project management method needs collaboration, communication, courage, focus on value, respect for people and a holistic approach to work. This runs counter to traditional western hierarchies and probably limits its usefulness in the West.

Contact Us

  • (+353)(0)1-443-3807 – IRL
  • (+44)(0)20-7193-9751 – UK
Saving Energy Step 1 ? Implementing a Management System

There has been much hype down the years regarding whether management is art or science. Thankfully, where people are concerned the pendulum has swung away from standard times in sweatshops in the west. However, when it comes to measuring physical things like harvest per square meter and the amount of energy consumed there is no substitute for scientific measurement, and this implies a system.

Managing energy cost and consumption down is like any other strategy. American engineer / statistician / management consultant W. Edwards Demming may have passed on in 1993. However he was as right as ever when he said:

  1. When people and organizations focus primarily on quality, this tends to increase and costs fall over time.
  1. However, when people and organizations focus primarily on costs, costs tend to rise and quality declines over time.

Demming believed that 90% of organizational problems arise from systems we put in place ourselves. This can be because we are so accustomed to them that we fail to notice when they are no longer relevant. The currently prevailing laissez faire towards energy is a case in point. What is managed improves and what is not, deteriorates. We know this. Let us take a look at how to apply this principle to energy management.

First, you need to get the subject out the closet and talk about it. How often do you do this is your boardroom, and how does energy rank against other priorities? Good governance is about taking up a position and following through on it. Here is a handy checklist you may like to use.

  • Do we use a consistent language when we talk about energy? Is it electricity, or carbon emitted (or are we merely fretting over cost).
  • How well engaged are we as a company? Looking up and down and across the organization are there points where responsibility stops.
  • How well have we defined accountability? Do we agree on key performance areas and how to report on them.
  • Are we measuring energy use at each point of the business? When did we last challenge the assumption that ?we’re doing okay?.
  • Have we articulated our belief that quality is endless improvement, or are we simply chasing targets because someone says we should.

A management system is a program of policies, processes and methods to ensure achievement of goals. The next blog focuses on tools and techniques that support this effort.

Ready to work with Denizon?

Contact Us Today