Are Master Data Management and Hadoop a Good Match?

Master Data is the critical electronic information about the company we cannot afford to lose. Accordingly, we should sanitise it, look after it, and store it safely in several separate places that are independent of each other. The advent of Big Data introduced the current era of huge repositories ?in the clouds?. They are not, of course but at least they are remote. This short article includes a discussion about Hadoop, and whether this is a good platform to back up your Master Data.

About Hadoop

Hadoop is an open-source Apache software framework built on the assumption that hardware failure is so common that backups are unavoidable. It comprises a storage area and a management part that distributes the data to smaller nodes where it processes faster and more efficiently. Prominent users include Yahoo! and Facebook. In fact more than half Fortune 50 companies were using Hadoop in 2013.

Hadoop – initially launched in December 2011 ? has survived its baptism of fire and became a respected, reliable option. But is this something the average business owner can tackle on their own? Bear in mind that open source software generally comes with little implementation support from the vendor.

The Hadoop Strong Suite

Free to download, use and contribute to

Everything you need ?in the box? to get started

Distributed across multiple fire-walled computers

Fast processing of data held in efficient cluster nodes

Massive scaleable storage you are unlikely to run out of

Practical Constraints

There is more to Hadoop than writing to WordPress. The most straightforward solutions are uploading using Java commands, obtaining an interface mechanism, or using third party vendor connectors such as ACCESS or SAS. The system does not replace the need for IT support, although it is cheap and exceptionally powerful.

The Not-Free Safer Option

Smaller companies without in-depth in-house support are wise to engage with a technical intermediary. There are companies providing commercial implementations followed by support. Microsoft, Amazon and Google among others all have commercial versions in their catalogues, and support teams at the end of the line.

Check our similar posts

Spreadsheet Woes – Burden in SOX Compliance and Other Regulations

End User Computing (EUC) or end User Developed Application (UDA) systems like spreadsheets used to be ideal ad-hoc solutions for data processing and financial reporting. But those days are long gone.

Today, due to regulations like the:

Sarbanes-Oxley (SOX) Act,
Dodd-Frank Act,
IFRS (International Financial Reporting Standards),
E.U. Data Protection Directive,
Basel II,
NAIC Model Audit Rules,
FAS 157,
yes, there?s more ? and counting

a company can be bogged down when it tries to comply with such regulations while maintaining spreadsheet-reliant financial and information systems.

In an age where regulatory compliance have become part of the norm, companies need to enforce more stringent control measures like version control, access control, testing, reconciliation, and many others, in order to pass audits and to ensure that their spreadsheets are giving them only accurate and reliable information.

Now, the problem is, these control measures aren’t exactly tailor-made for a spreadsheet environment. While yes, it is possible to set up a spreadsheet and EUC control environment that utilises best practices, this is a potentially expensive, laborious, and time-consuming exercise, and even then, the system will still not be as foolproof or efficient as the regulations call for.

Testing and reconciliation alone can cost a significant amount of time and money to be effective:

It requires multiple testers who need to test spreadsheets down to the cell level.
Testers will have to deal with terribly disorganized and complicated spreadsheet systems that typically involve single cells being fed information by other cells in other sheets, which in turn may be found in other workbooks, or in another folder.
Each month, an organisation may have new spreadsheets with new links, new macros, new formulas, new locations, and hence new objects to test.
Spreadsheets rarely come with any kind of supporting documentation and version control, further hampering the verification process.
Because Windows won’t allow you to open two Excel files with the same name simultaneously and because a succession of monthly-revised spreadsheets separated by mere folders but still bearing the same name is common in spreadsheet systems, it would be difficult to compare one spreadsheet with any of its older versions.

But testing and reconciliation are just two of the many activities that make regulatory compliance terribly tedious for a spreadsheet-reliant organisation. Therefore, the sheer intricacy of spreadsheet systems make examining and maintaining them next to impossible.

On the other hand, you can’t afford not to take these regulations seriously. Non-compliance with regulatory mandates can have dire consequences, not the least of which is the loss of investor confidence. And when investors start to doubt the management’s capability, customers will start to walk away too. Now that is a loss your competitors will only be too happy to gain.

Learn more about our server application solutions and discover a better way to comply with regulations.

More Spreadsheet Blogs

Spreadsheet Risks in Banks

Top 10 Disadvantages of Spreadsheets

Disadvantages of Spreadsheets – obstacles to compliance in the Healthcare Industry

How Internal Auditors can win the War against Spreadsheet Fraud

Spreadsheet Reporting – No Room in your company in an age of Business Intelligence

Still looking for a Way to Consolidate Excel Spreadsheets?

Disadvantages of Spreadsheets

Spreadsheet woes – ill equipped for an Agile Business Environment

Spreadsheet Fraud

Spreadsheet Woes – Limited features for easy adoption of a control framework

Spreadsheet woes – Burden in SOX Compliance and other Regulations

Spreadsheet Risk Issues

Server Application Solutions – Don’t let Spreadsheets hold your Business back

Why Spreadsheets can send the pillars of Solvency II crashing down

amazon.co.uk

amazon.com

Contact Us

(+353)(0)1-443-3807 – IRL
(+44)(0)20-7193-9751 – UK

What ISO 14001 Status did for Cummins Inc.

Cummins manufactures engines and power generation products, and has been a household name almost since inception in 1919. It sells its products in over 300 countries, through approximately 6,000 dealerships employing 40,000 people. Because its product line runs off fossil fuel it is under steady pressure to display a cleaner carbon footprint.

Cummins decided to go for the big one by qualifying for ISO 14001 certification. This is a subset of a family of standards relating to managing environmental impact while complying with all applicable legislation. In this sense, it is similar to the ISO 9000 quality management system, because it focuses on how products are produced (as opposed to how those products perform). Compliance with ISO 14001 was a doubly important goal, because it is part of the European Union?s Eco Management and Audit Scheme and fast becoming mandatory on suppliers to governments.

The qualification process follows the well-established principle of plan, do, check, act. It begins with gap analysis to detect materials and processes that affect the environment. This is followed by implementation of necessary changes affecting operations, documentation, emergency strategies and employee education. The third step involves measuring and monitoring performance. Finally, the project moves into a phase of ongoing maintenance, and continuous improvement as circumstances change.

In Cummins case, the project was almost worldwide and called for environmental, health and safety reporting throughout the organisation. The information was shared via a globally accessible document repository, and then processed centrally at the head office in Columbia, Indiana USA.

Measuring environmental performance almost inevitably has other benefits that make it doubly worthwhile. Speaking at the 2014 National Safety Council Congress after receiving the top award for excellence, Cummins chairman and ceo Tom Linebarger commented on a journey that was ?nothing short of amazing? yet wasn’t even a ?pathway to the finish line?.

?All of us feel like we have way more to do to make sure that our environment is as safe as it could be,? he added, ?so that our sustainability footprint is as good as it can be and that we continue to set more aggressive goals every year. That’s just how we think about it.? Linebarger concluded.

If you are taking your company on a journey to new heights of environmental excellence, then you should consider choosing ecoVaro as your travelling companion. We are environmental management specialists and have proprietary software geared to process your data. We also have a wealth of experience, and a treasure chest of roadmaps to help you achieve your goal.

Telemetry and the Survival of the Human Species

Without moisture, plants die. Without fodder, the animal food chain collapses. This is why climate change is the greatest threat humankind faces. Crop management needs timely information regarding ambient conditions, and also in the soil itself. In dry areas, online knowledge of trends in rainfall, sunlight, wind speed, leaf moisture, air temperature, relative humidity and solar radiation are indicators of soil stress that can be deadly for plants, and everything that relies on them.

As climate change bites, the need to find solutions accelerates. Drones swoop across to monitor ambient conditions, while probes sunk into plants and the earth in which they grow transmit information to big data repositories for feedback to administrators. In Australia, a remarkable cattle farmer is applying the same approach to his herds.

Nuffield scholar Rob Cook has always been on the edgy side of things. He lost his mobility in a helicopter crash in 2008 patrolling farmland but that has not deterred him. If anything, it has freed his mind to explore the potential that telemetry offers farmers in Australia. He shared this potential with the young beef producers in Roma Australia recently, and here is a summary what he said.

Being wheelchair bound he had to shift from herding with cattle dogs to a more scientific approach. He bought a farm 230 miles / 370 kilometres inland from Brisbane in a warm, temperate climate with significant rainfall even in the driest months. He uses observant software that reports on critical issues like water levels indicating animal consumption, and supplementary water flows from a central irrigation channel.

He also monitors fodder sources for dryer months, and moisture levels in food stocks. Rob is committed to making every blade of grass count. ?We even have the ability to take a photo of the cattle when they are taking a drink of water,? he explains, and that provides valuable information regarding tick and fly infestation and overall condition.

None of this would be possible for Rob Cook without telemetry, which is the process of collecting data at remote points and transmitting it to receiving equipment for analysis. Independent farmers do not have equipment to fund these analytic resources on their own, and use big data resources in a cloud to obtain reports. ecoVaro is on top of current trends. Please speak to us when you need independent advice.