Original status: 1-Dispatched; Suggested Status: NEW


Original submitter: davidvc

Description:
Right now if you open a database connection to view its tables, the metadata for
*all* the tables is pulled over, and Creator hangs until this completes.  This
is fine for databases with a small number of tables, but for a database with a
large number (an internal user has 11,000 tables in their Sybase database), it's
a killer and is a waste of time and memory resources.

We should have a way to let the user limit what table metadata is pulled in. 
Also, we should probably not hang the system while processing all results but
display the first set of results immediately while continuing to process the
remainder.

Workaround:
I agree this is a problem.  I think it also indicates a series of 
'scalability' tests we should probably think through.  What if a table 
has 10,000 columns, for example?

here are some thoughts, none of which I have tested or investigated in 
much detail...

For the 11,000 table problem, the quick workaround is to extract the 
table definitions for the tables he cares about and import them into a 
Java DB database.  He can do his design there, and then when he's ready 
to test with "live" data, switch the data source to be Sybase instead of 
Java DB (although I'm not sure myself how you do this...)

A possible longer-term solution is to allow the user to apply a filter 
pattern connection that can then be used to select a subset of the 
tables.  Another approach is, we don't have to load the entire result 
set into memory before displaying tables.  We can display the first 20 
or so while we're still loading the rest in the background.

We should also keep an eye on memory consumption, caching the metadata 
in a file if the result set is too big.

David VC

----------------
Its been running for over 20 hours so far and it still is not half way 
done.  If it goes through all the views after the tables then it is 
going to take even longer.

I am going to leave it up over the weekend and see if the IDE eventually 
comes back or just runs out of memory and crashes.

- xxxx@yyyy.zzz