My idea is to export primary library resources as unique files. E.g.
any module can then easily add a sibling file with their private
information.

Modules can also use a database. It's not library reposnsibility to be
container itself. Library is container of references.

I hope above assumption (that library roots can be mapped fo files) is
correct. I bet on javac inability to take its input from non-files.

I think I talked about the "references", too (in a clumsy way, you
know). If the resource is a simple thing (e.g. jarfile, folder), then
a single .xml can easily describe it.

I would like to point out that the library must be standalone
selfcontained entity. It may be stored either in the project or
locally in IDE. Thus it must not store any of its part e.g. in project
settings context or elsewhere. 

From the API point of view the Java Libraries API shouldn't depend on
the projects infra nor the project specific java API, it should only
depend on the generic classpath API. (BTW, Svato, what do you thing
about this separation? The ProjectPath is now mixed in the generic
classpath API, shouldn't we separate it to o.n.api.java.project
package? I know this is kind of paranoia, but we should strictly avoid
circles in our API dependencies.)

Petre, please consider making the libraries API a separate autoload
module.

I am sorry if repeating something what you already are aware of.

Library provides an API to query contained resources, see issue
#23828. Resource providers must follow an SPI protocol to enable it.

Something like: 

interface VolumeProviderSPI {
  /** Hooks allowing to enhance libray by
   secondary content. */
  void notifyVolumeAdded(Volume)
  void notifyVolumeRemoved(Volume)
  void notifyVolumeChanged(Volume)  

  // ... or pull approach?
}

where: 

interface VolumeSPI {
  Class getValueClass()
  Object getValue()
  Library getLibrary()
  VolumeProviderSPI getProvider()
  bool isValid()
}

How would be these interfaces used ?
What is the client-side API to access the data inside the library ?

I see that one can call VolumeSPI.getProvider().notifyVolumeAdded() --
does it mean that the provider can be instructed to add volume this way ? 

For a Provider, I still do not see a way how to store information
within the library (I expected that I will be able to store a
FileObject inside)

A counter-proposal:
What about creating a simple Lookup created from providers registry as
the client-side API, give the Providers the storage folder so they can
extract and store their data ? Then you would need only an interface
to create a Provider instance (these would be placed into the
Providers registry) and another interface to bind a Provider instance
to a particular library and its storage (c.f. new DataLoaders and
their lookup providers, or Java Platforms and services registered in
there).
Client API would be then simply a plain Lookup.Provider.
Stuff like reporting changes in library contents could then be done at
Provider-specific protocol instead.

I forgot to advocate why I believe in storing provider's data (e.g.
references to the real resources) within the library storage:
The provider then does not need to monitor library's lifetime and
clean-up when the library is removed. That works even in the case when
 not all providers are available (e.g. disabled modules) at the time
the library is deleted from the system.

Thanks for hints. I'll publish whole non-simplified design (an XMI) soon.

Back to original request (library storage SPI part). It can be
achieved just by having resource identity.

Some providers such as completion database provider stores their
secondary data even for non-library resources. That provider has
implemented data invalidation logic. It does not care whether primary
resource is part of library or not.

I need more opinions as it looks like any resource type can be used
directly.

If the above SPI simplificatiom proves to be true then I remove the
SPI storage part.

Tomas Z. raised additional requirement. Content of used library must
be exposible using ContentDescriptors. To create ContentDescriptor one
need library reference ProjectMember. To achive it there must be two
lookup content providers layers:

   - library reference in project (one Lookup per referenece) NEW
   - library definition (one Lookup per library definition)

The library reference layer is optional for all providers if they do
not want to provide ContentType queriable resources.

My recent understanding is that the library reference layer would
duplicate another wheels of project infrastructure.

I decided to create FilterLookup.expandResult() transformation lookup
support that can be easily used by ProjectEnvironmentProviders. I'll
mark library resources by a kind of LibraryReferenceContentDescriptor
so providers can identify library references easily.

What is the FilterLookup good for?

It simplifies lookup providers that do lookup transformations. You
just white Convertor and it cares for other issues. 

In particular it allows to write ProjectEnvironmentProvider sensitive
to library references so interested parties can add their project
dependend stuff (e.g. JavaDescriptor). 

  getLookup(ProjectMember pm) {

     if (pm.getLookup().lookup(LReferenceDescriptor.class) != null) {
        Lookup.Template t = 
            new Lookup.Template(ClasspathVolume.class);
        final ProjectMember mypm = pm;
        return FilterLookup(pm.getLookup(), t, new Convertor() {
           Object convert(Object obj) {
               Lookup.Item item = (Lookup.Item) obj;
               ClasspathVolume volume = item.getInstance();
               add_to_lookup(JavaDescriptor(mypm, volume));
           }
        });
     }
     return_lookup();
  }

Still some boilerplate code survived....

Sorry, but this will not work. The PM.getLookup() == Lookup.EMPTY when
called from inside the PEP.createLookup method.

I see. It almost eliminates pure PEP based solution.

I can introdude special purpose hook called from library reference PEP
implementation:

   interface LibraryReferenceProjectEnvironmentProvider {
      /**
       * Exactly the same contract as PEP just with
       * additional referenced library parameter.
       */
      Lookup getLookup(ProjectMember theReference, Library lib);
    }




Besides FilterLookup solves infinite recursions among several
independent providers the same you avoid by passing cripled PM with
empty lookup.

ReferenceEnvironmentProvider hook added. I hope it enables Tom to
write real classpath resource support.

I think the API is ready for next review.

The projects prototype has been canceled. For more details see
http://www.netbeans.org/servlets/ReadMsg?msgId=619519&listName=nbdiscuss