DW Cloud - Local Data Connector - 1000 Line Limit is BAD THING
It's common to reference a Local Data Connector data source and have the Autoindex routine be set to create a data record if there is no matching record. Especially common in verifying a migration to find items that got missed. It's invariable that some docs won't come over, and this is a great way to find them. It's common for these to sometimes be a ton of records(often > 1,000,000). In 7.3 the system will only use 1,000 records from the remote data at a time, even if the query limit in the LDC config file is set to 10,000 rows. This limit causes a huge and unnecessary expenditure of time.
Can we get that restriction removed?
Thanks!
![](https://secure.gravatar.com/avatar/29eccbfe55bb2edb3739aa96f154080d?size=40&default=https%3A%2F%2Fassets.uvcdn.com%2Fpkg%2Fadmin%2Ficons%2Fuser_70-6bcf9e08938533adb9bac95c3e487cb2a6d4a32f890ca6fdc82e3072e0ea0368.png)
-
PEGS commented
A datasource created with an ftp file (Cloud) does not have the 1000 record limitation,but exporting the data periodically is a pain.
-
Zach Heser commented
At least increase the limit to 10,000. This should have a minimal impact on performance.
I currently have 50 Autoindex jobs set to reference a master vendor record of 50,000 vendor accounts. If the limit was 10,000 I would only need 5 AI jobs.
Alternatively, you could add the ability to add data records through workflow where we could loop through the records 1,000 at a time.