Playing Devils AdvocateAugust 17 2011 at 9:18 AM
No score for this post
from IP address 188.8.131.52
Response to Multi File Load Best Practice
If the files were created correctly and did not require rules files to interpret - this could be a very fast and efficient way to load data from disparate sources. You could quite easily build a script that loops through all files in a directory and load them so they do not have to be called out by name and you do not have to worry if the name changes.
That being said - that is probably not what has been done - and I am guessing you cannot do that.
*** So the pains that you will feel are due to:
********Potential Number of Rules File Objects to Maintain
********Inability to automate efficiently (do you have to change automation if get more?)
********Inability to debug bad data or data issues if validation finds an issue
********Inability to create an efficient way to validate to the source(s)
So what is creating this? Where do the files originate? Are they actually being created by IT - or is this something like users sending in files from all over? The source of these files definitely changes the picture because let's say they are coming from users - then it is not just a simple - slap it all in a relational db and go. Now you have to create a process and application to gather the data to that relational db. If this is a user owned app - like by Finance - then maybe they do not have a relational db or relational db expertise - so you have to involve IT - which they may not want to do... If you are getting pushback - I am guessing it is because of something like this that would cost $ to implement a new solution and they want to take a short cut to avoid that cost, resources needed, etc.
I do not think you will not find a white paper that says loading flat files is wrong - because flat files load faster than anything else... and I doubt you find a white paper that says to always source from a relational db warehouse, datamart or data store - because the early glory of essbase was in the fact that it could source from nearly anything.
But what you may want to do is document the risks they are assuming - and the future pains they may feel - by choosing to go with this solution - and offer up better solutions/options and why those solutions make sense to initiate now - even if they do have some $$ associated to them - which they undoubtedly will.
GOOD LUCK with this one.
- Re: Playing Devils Advocate - Anonymous on Aug 17, 4:28 PM
- Loading many flat files - Alan Hutchins on Aug 18, 11:18 AM
- So much for IT driving good process - Cameron Lackpour on Aug 18, 11:46 AM
- Re: So much for IT driving good process - Anonymous on Aug 18, 8:46 PM