Thanks for the kind words! Agree, we are really proud of the continuous retro funding work we did last year.
Our REST/GraphQL API crawler factories enable fine-grained options on batch size, pagination, rate limit, and back-off policies. They work well, but are very difficult for a casual non-technical person to configure well, since it requires intimate understanding of how an API works to configure correctly. You can find all instances we have here:
Ideal future, we'd love to be able to auto-discover these parameters (e.g. grow incremental, exponential back-off similar to how TCP works), but we're not there yet.
Thanks for the kind words! Agree, we are really proud of the continuous retro funding work we did last year.
Our REST/GraphQL API crawler factories enable fine-grained options on batch size, pagination, rate limit, and back-off policies. They work well, but are very difficult for a casual non-technical person to configure well, since it requires intimate understanding of how an API works to configure correctly. You can find all instances we have here:
https://github.com/opensource-observer/oso/tree/main/warehouse/oso_dagster/oso_dagster/assets/default
Ideal future, we'd love to be able to auto-discover these parameters (e.g. grow incremental, exponential back-off similar to how TCP works), but we're not there yet.