Hi,
I have a potential software project for a large ad composition agency. The requirement is briefly as follows:
1. There will be front offices located in more than a dozen locations across multiple states. These front offices would be posting requirements about advertisements to multiple back office located in some other country. Each front office would have about 10 users.
2. Each ad could be approximately 6MB in size consisting of metadata and graphic files.
3. There will be 3-4 back offices with about 200 users in each location.
4. Jobs posted by the front offices would then be picked up by the back office personnel for design and quality checks. There is a separate workflow involved here with various rules about designer and quality check allocation etc. There is a heavy demand for a lot of asynchronous backend processing like automatic quaility check of documents, PDF generation, printing etc. The final ad would then be submitted which will then be received by the respective front office.
5. The efficiency, robustness and scalability of the solution is critical as the data transfer is going to be huge (about 2000 ads per day per front office which translates to about 150000 MB per day!). We can imagine the amount of load on the back office servers with this kind of data transfer.
6. The response time of the system is absolutely critical as turnaround time for an ad is really fast. Ads posted today for composition need to be ready for printing the next day!
7. As we can see, an absolutely efficient and robust backend scheduling/queueing engine is the core requirement.
The project is in the initial stages and I have been thinking of employing a solution architecture using
1. IIS (as web servers in each front & back office location)
2. SQL Server 2000 (as the database with redundancy and recovery features)
3. Multiple MSMQ Server installations (for proper prioritisation of jobs and queueing to be picked up by the appropriate back office + forwarding backend processing requirements to auto processing servers for quality check of documents, PDF generation, printing etc.)
4. Multiple BizTalk Server 2004 installations (possibly a BizTalk Server group for each back office location)
5. Backend processing servers
Any thoughts on the viability of such a solution architecture or on the technology selection would be really great. Thanks.
Need help on solution architecture employing BizTalk Server & MSMQ Server
Sideshow Paul
I do not know if you have read this:
http://msdn.microsoft.com/library/default.asp url=/library/en-us/BTS_2004WP/html/922d9b1a-c6f2-4d42-9f7a-2876eac7f50d.asp
It’s a performance doc. I think you could centralize a little BizTalk farm if you don't have so much traffic, as i think. BizTalk is more powerful than it looks, and expensive :)
Brian Hartman - MSFT
Thanks.
Agreed. My initial thinking on the architecture is to pass only xml metadata through the system required by the BizTalk message receivers. The actual work documents could be handled separately via FTP Servers/network share (over VLAN)/SAN/NAS. This approach will drastically reduce the load on the queueing/BizTalk servers.
I am considering a single centralized server farm for the front offices (located in the same country) and one centralized server farm for the back offices (located in the same country). These two server farms would be linked up by a dedicated 2 mbps Internet link.
Another option is to scale down the network infrastructure and position at each front office location which then interact individually with the back office server farm.
Any thoughts on these two approaches are welcome.
For the front office server cluster I am considering tiers in the following order
Firewalls -> Network load balancing (to balance the load among the web servers) -> IIS farm (for processing new job requests from the front office + receiving finished jobs back from the back office etc.)-> MSMQ cluster (to queue the jobs based on priority and also provide loose coupling) -> BizTalk Server group (message receive functions which receive only the metadata) -> BizTalk Server group (processing functions which will process the actual job documents viz creating the work package, compression etc). The BizTalk Server group for processing functions will be enabled with load balancing. These work packages are then forwarded to the back office file share for further processing.
I have some initial network infrastructure diagrams. If you could provide an email address I could forward them to you. It will help in delivering a clearer picture and we can discuss further. You may write to me at meghanathsingh@hotmail.com. Your help will be highly appreciated.
Thanks
bldink
You write:
"My initial thinking on the architecture is to pass only xml metadata through the system required by the BizTalk message receivers. The actual work documents could be handled separately via FTP Servers/network share (over VLAN)/SAN/NAS. This approach will drastically reduce the load on the queueing/BizTalk servers."
Skip the ftp servers, you don't need them, and they will probably cause more problems than they will solve. There is no problems posting messages in the 10MB size you're mentioning through webservices and then into biztalk. I've a site running receiving some 500 messages over a span of a few days each month, each message between 5 and 20 MB, through Biztalk and into a SQL server repository. I've had no problems with the load on the BTS server (only using one).
The MSMQ also has a limit on messages of 4MB, but Biztalk messages can be any size. There is a new message queue module called MSMQ-T, shipping with BTS, which breaks this limit. You can also use this alone, see http://blogs.msdn.com/darrenj/archive/2004/03/31/104766.aspx for more info on this.
I would go with webservices as ports for receiving the messages, or possibly MSMQ-T ports, although there is no difference as long as the webservice push the message into BTS. BTS can and will handle this with efficiency. Just watch a few points: The .net framework used by the IIS must be configured for large posts through the webservices, configured through f.e. machine.config, also change the timeout limit up from the default.
Dan W Hollingsworth
Few observations (note that they are based on the very limited understanding of your requirements - base on your post)
it seems that while a lot of data is being moved back and forth - it might not be needed to handle all the data "inside" the system - only the metadata (e.g. once the data arrives at the backoffice store the data on a NAS and then pass the meta-data in the different workflows of the backofffice )
You specified some of the performance requirements - there are many other considerations that can affect the architecture in areas of Security, Flexibility, availability, scalability etc.
You've mentioned few servers that you are going to use - sound as good as any list for that matter. The questions is how are you going to use them what data flows will you have why do you want IIS maybe it makes more sense to use VSTO and embed your application as part of their outlook applications (http://msdn.microsoft.com/office/understanding/vsto/default.aspx pull=/library/en-us/odc_vsto2005_ta/html/OfficeVSTOIntroducingOutlookAdd-in.asp). While you are at it why not use SQL Server 2005 (you'd also have an additional option for reliable messaging - service broker, which solves some problems MSMQ has with in-order delivery)
You probably want to se-up a proof of concept as quickly as possible to test your assumptions on performance etc. you can use tools (like Shunra Storm http://www.shunra.com/content.aspx pageId=68 - to simulate the WAN)
HTH,
Arnon
Silliea
To decide on these alternatives you need to consider several aspects for example:
What are your availability requirement for front offices how reliable will their connection to the centralized server farm what are the budget constraints what are the security implications of having servers at each front office etc.
You can look at my presentation on SEI's Architecture Tradeoff Analysis Method (ATAM) for a method to examine such alternatives
Arnon