DW Cloud - Local Data Connector - 1000 Line Limit is BAD THING
It's common to reference a Local Data Connector data source and have the Autoindex routine be set to create a data record if there is no matching record. Especially common in verifying a migration to find items that got missed. It's invariable that some docs won't come over, and this is a great way to find them. It's common for these to sometimes be a ton of records(often > 1,000,000). In 7.3 the system will only use 1,000 records from the remote data at a time, even if the query limit in the LDC config file is set to 10,000 rows. This limit causes a huge and unnecessary expenditure of time.
Can we get that restriction removed?
Thanks!
Chris McFarland
shared this idea
-
Zach Heser
commented
At least increase the limit to 10,000. This should have a minimal impact on performance.
I currently have 50 Autoindex jobs set to reference a master vendor record of 50,000 vendor accounts. If the limit was 10,000 I would only need 5 AI jobs.
Alternatively, you could add the ability to add data records through workflow where we could loop through the records 1,000 at a time.