I had no luck sorting large files with Sort transformation. 1 or 2 GB are still so-so, but 3 - 5 GB seem to hang up and not finish.. Anyone else encounter this What would you recommend as workaround (maybe some other program)... Thanks.
How do you sort large files
Hot Topic
- Wishlist: MOST WANTED Tasks, Sources, Transformations, and Destinations?
- How to enable remote errors for Reporting Services 2005
- SQL Server 2005 Sept CTP - Failed Installation - It Wants Disk 2????
- ODBC Destination in SSIS
- Data mapping to column 'CustomerID' was already found in the data. Make sure that no two schema definitions map to the s
- It doesnt execute my assemblies.
- More problems with activation
- database maintenance plan - reorganize data pages and indexes
- Excel connection error The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error
- Error Runing a subscription on report server 2000
SQL Server
- Transaction (Process ID 58) was deadlocked on thread error...
- Flat File Source Column Parsing Error
- Script Table with secondary indexes
- Acquire Connection Execution
- How to launch programatically a SSIS package?
- An error occurred while receiving data: '10054(An existing connection was forcibly closed by the remote host.)'.
- Deleting Database contents - Microsoft SQL Server Management Studio Express
- Business Scorecard Manager Export ro RTM SSRS
- issqluser: how is it set/unset
- Passing parameter using .SQL
How do you sort large files
TFSFailedInstall
The reason I asked is I also had been working on a scalable sort and am curious what folks are using so far. I just finished. My sort is 100% plug and play compatible with stock Sort tranformation. It's about 3x faster on small data sets (< 1 GB) and orders of magnitude faster on large (> 1 GB). Everyone invited to download from www.ivolva.com and send feedback.
RolandM
I downloaded your sort component. But, it's cancelling my job when it get's to the new ExtraSort component of yours. I will say that the component is reallly easy to use and install, that is fantastic. But, why is this cancelling my job
nickyG
This isn't surprising really. You are limited by the maximum addressable memory by a SSIS package which is 2GB (or 3GB with the 3GB switch in boot.ini).
The sort transformation is done in memory so, likely that SSIS is doing memory swapping. You can check this out using the SQL Server : SSIS Pipeline perfprmance monitor object
Some useful links here regarding memory usage:
http://www.scalabilityexperts.com/default.asp action=article&ID=167
http://blogs.conchango.com/jamiethomson/archive/2005/05/29/1486.aspx
A link regarding SSIS perfmon:
http://blogs.conchango.com/jamiethomson/archive/2005/06/09/1583.aspx
-Jamie
Steve_Hall
Go to http://www.ordinal.com to try out their Sort component. In my experience it has been really performant for large data sets. Some of our early adopters are using it as well.
Regards,
ash