Advice on a process

I'm looking for best-practices advice. I have a flat file that gets produced by a mainframe system every day. I need to import this data into a relational database each night. I will use the data to do some numerical optimizations (generallly it will be a knapsack algorithm) for scheduling of machine and personnel resources. The file is a "rolling" list of our back orders. By rolling, I mean that when a job is complete it drops off the list and when a new order comes in it appears on the list. I have all the data I need to do the optimizations contained in the flat file. As you can imagine, the data in the flat file is highly redundant and not in any normal form. I've successfully made a package that imports the flat file from the mainframe. Now, I'm ready to make a package that parses the flat file and stores the information in it in a normalized form while taking account of the "rolling" nature of the data.

I need to do differing things. If I get a new order, I need to add it to the order file. If I get an order that has changed (which I can know by comparing a modified date in the flat file to a modified date in the order table), I need to modify the entries for that order. If an order for a product that isn't already on file shows up, I need to add the product to the product table. I also need to look at the orders that are already on file and see which ones are no longer in the nightly import so I can flag them as being closed. In any event, you get the drift.

My normal habit would be to write a program that does these things. I could certainly do that in this case and run it as an external process in my package. Given the tool SSIS is, is a special purpose program a reasonable approach to take within the context of SSIS. Or is there some better approach using some of the new tools that SSIS makes available. I'm certainly more comfortable writing a special purpose app but would prefer to take a best practices approach to this kind of problem.

Thanks in advance for advice.




Answer this question

Advice on a process

  • dfgdsg

    Thanks for the feedback. Your article you linked to is extremely helpful.



  • Balaji K

    It seems you need to do the following:

    1. Determine if an order is new or not (here's how you do that: http://www.sqlis.com/default.aspx 311)
    2. Determine if an order is changed by comparing data fro 2 heterogenous sources
    3. Determine if any products are new or not
    4. Compare the target with the contents of the file to detemine which are "closed"

    SSIS is well suited to all of these operations. In my wholly unbiased opinion (!!!) it would be folly to attempt to do this any other way when you have a tool like SSIS available. Just my two-penneth worth.

    -Jamie



  • Advice on a process