[ESIP-CF] Esip-cf Digest, Vol 5, Issue 1 - Thoughts On Groups vs. Compound/var len arrays
Joe.Glassy
joe.glassy at ntsg.umt.edu
Wed Feb 1 13:21:59 EST 2012
Hello Ted, et al
See responses interspersed below..
On Wed, Feb 1, 2012 at 10:26 AM, Ted Habermann <ted.habermann at noaa.gov> wrote:
> Upendra et al.,
>
> I agree that User-Defined Types are a potentially very interesting approach
> to adding structure to metadata in granules. One of the challenges is that
> UDTs have been developed as a tool for describing data structures instead of
> attribute structures. In other words, we would need to write metadata into
> variables instead of into attributes.
>I suspect that this would change the
> mechanisms we use to write and extract this information considerably...
This may be true... or perhaps not, and certainly worth
discussing. The way I see it, if expressing a UDT as residing in a
'variable', it would be no more or less complex to de-reference in a
'variable' vs. an 'attribute. As a community, we've gotten somewhat
accustomed to tightly associating 'metadata' with the particular
mechanism known as an 'attribute' in HDF4/5, but I see that as an
accidental habit more than any sort of necessity. In particular, this
is one reason I've long advocated the use of HDF5 'groups' as a more
formal protocol for delineating what is 'data' and what is 'metadata'
within the scope of a granule. For example, if one stores all
metadata in a group called "/METADATA" and all 'data' under a group
called "/DATA", then this easily communicates to both producers and
consumer what is where. Then, if a UDT instance is expressed as a
'variable' , residing under "/METADATA", most of the historical
distinction between attribute and variable goes away, and we all get
what we need - a way to express metadata that just happens to be
structurally rich, in a variable. Just some food for thought.
best,
Joe
> I have discussed the question of arrays of attributes with Mike Folk from
> the HDF group (cc'ed) recently, and plan to follow up on that approach... If
> we could extend the idea of UDT's into the attribute space it would be very
> cool!
>
> Ted
>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 31 Jan 2012 15:58:38 -0500
>> From: Upendra Dadi <upendra.dadi at noaa.gov>
>> To: esip-cf at lists.esipfed.org
>> Subject: Re: [ESIP-CF] CF Cluster telecon today
>> Message-ID: <4F2855FE.7000500 at noaa.gov>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hi All,
>> I added a link on the Wiki to a netCDF file created by NDBC which
>> uses groups.
>>
>> On a related note, I want to ask this group if they have considered
>> using user defined structures in place of groups. As you might be
>> knowing that netCDF-4 allows compound types which are based on C struct.
>> Compound types along with variable length arrays could often be a good
>> substitute for using groups.
[No.. I find this assertion highly suspect as I read it. Compound
types along with variable length arrays, I think, serve a distinctly
different conceptual role than 'groups'; e.g. both have unique value
independent of each other.
With regard to CF, incorporating groups, I
>> am guessing, is much more challenging than including compound types.
>> Anybody disagrees?
YES
I am looking for some examples where groups are much
>> better suited than using compound types and/or variable length arrays.
Again, I hope I'm not jumping too far into a prior discussion with a
lot of context I'm missing... but... I see groups as a fundamental and
critically useful organizing mechanism, that also happens to be a
communication mechanism to both producers and consumers of data. I see
'compound types and/or variable length arrays' simply as more
sophisticated, structurally rich containers of data, but I believe
their role is still to function more as 'data containers' which is
different than the role of 'classification mechanism' that groups
play. I do realize this is a philosophical or 'style' distinction,
but ultimately that's how day-to-day practices are formed and
reinforced through wide adoption. Just some thoughts...
>> Please let me know if you have one. Thank you!
>>
>> Upendra
>>
>> On 1/31/2012 7:11 AM, Raskin, Rob (388M) wrote:
>> > This is a reminder that the rescheduled CF Cluster telecon is today:
>> >
>> > When: Tuesday Jan 31, 2pm ET / 11am PT
>> > Dial: 877-668-4493
>> > Access code: 23138379
>> >
>> > Agenda:
>> > -HDF-EOS / CF Conversion Table
>> > -Summer Meeting Technical Workshop showcasing CF-aware tools
>> > -CF extensions desired at Level 1, Level 2, and Level 3
>> > -Mapping to ontology
>> >
>> > http://wiki.esipfed.org/index.php/CF
>> >
>> >
>> > ------------------------------------
>> > Rob Raskin
>> > Group Supervisor, Science Data Engineering and Archiving
>> > Instrument Software and Science Data Systems Section
>> > Jet Propulsion Laboratory
>> > Pasadena, CA 91109
>> > (818) 354-4228
>> > _______________________________________________
>> > Esip-cf mailing list
>> > Esip-cf at lists.esipfed.org
>> > http://www.lists.esipfed.org/mailman/listinfo/esip-cf
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 31 Jan 2012 16:31:37 -0700
>> From: Russ Rew <russ at unidata.ucar.edu>
>> To: Upendra Dadi <upendra.dadi at noaa.gov>
>> Cc: esip-cf at lists.esipfed.org
>> Subject: Re: [ESIP-CF] CF Cluster telecon today
>> Message-ID: <10347.1328052697 at unidata.ucar.edu>
>>
>> > Hi All,
>> > I added a link on the Wiki to a netCDF file created by NDBC which
>> > uses groups.
>> >
>> > On a related note, I want to ask this group if they have considered
>> > using user defined structures in place of groups. As you might be
>> > knowing that netCDF-4 allows compound types which are based on C struct.
>> > Compound types along with variable length arrays could often be a good
>> > substitute for using groups. With regard to CF, incorporating groups, I
>> > am guessing, is much more challenging than including compound types.
>> > Anybody disagrees? I am looking for some examples where groups are much
>> > better suited than using compound types and/or variable length arrays.
>> > Please let me know if you have one. Thank you!
>>
>> Groups and compound types are intended for different purposes. A group
>> is really just a name space or container for variables, dimensions,
>> attributes, user-defined types, other groups, and data. So, for
>> example, it would be possible to store data for separate geographic
>> regions in distinct groups, in which the same names could be used for
>> all the data objects, e.g.:
>>
>> http://www.unidata.ucar.edu/netcdf/workshops/2010/netcdf4/Groups.html
>>
>> Groups can be useful for factoring out other kinds of common information
>> than geographic regions, e.g. shared metadata.
>>
>> Compound types, like structs in C and derived types in Fortran, are
>> useful for specifying data that has named members, possibly of different
>> types. For example
>>
>> netcdf odnc4 { // obs data in netCDF-4, inspired by example from Tom
>> Kunicki
>> types:
>> compound observation_t {
>> int time ;
>> float temperature ;
>> float pressure ;
>> }; // observation_t
>>
>> compound observation_att_t {
>> string time ;
>> string temperature ;
>> string pressure ;
>> }; // observation_att_t
>>
>> dimensions: // these two unlimited dimensions are
>> independent
>> station = UNLIMITED ;
>> observation = UNLIMITED ;
>>
>> variables:
>> int station_id(station) ;
>> station_id:standard_name = "station_id";
>> float latitude(station) ;
>> latitude:units = "degrees_north" ;
>> float longitude(station) ;
>> longitude:units = "degrees_east" ;
>> float elevation(station) ;
>> elevation:units = "feet" ;
>> elevation:positive = "up" ;
>> observation_t observations(station, observation) ;
>> observations:coordinates =
>> "observations.time longitude latitude elevation" ;
>> observation_att_t observations:units =
>> {"days since 1929-1-1 0:0:0", "degF", "hectopascal"} ;
>> ...
>>
>> --Russ
>>
>> > Upendra
>> >
>> > On 1/31/2012 7:11 AM, Raskin, Rob (388M) wrote:
>> > > This is a reminder that the rescheduled CF Cluster telecon is today:
>> > >
>> > > When: Tuesday Jan 31, 2pm ET / 11am PT
>> > > Dial: 877-668-4493
>> > > Access code: 23138379
>> > >
>> > > Agenda:
>> > > -HDF-EOS / CF Conversion Table
>> > > -Summer Meeting Technical Workshop showcasing CF-aware tools
>> > > -CF extensions desired at Level 1, Level 2, and Level 3
>> > > -Mapping to ontology
>> > >
>> > > http://wiki.esipfed.org/index.php/CF
>> > >
>> > >
>> > > ------------------------------------
>> > > Rob Raskin
>> > > Group Supervisor, Science Data Engineering and Archiving
>> > > Instrument Software and Science Data Systems Section
>> > > Jet Propulsion Laboratory
>> > > Pasadena, CA 91109
>> > > (818) 354-4228
>> > > _______________________________________________
>> > > Esip-cf mailing list
>> > > Esip-cf at lists.esipfed.org
>> > > http://www.lists.esipfed.org/mailman/listinfo/esip-cf
>> >
>> > _______________________________________________
>> > Esip-cf mailing list
>> > Esip-cf at lists.esipfed.org
>> > http://www.lists.esipfed.org/mailman/listinfo/esip-cf
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> Esip-cf mailing list
>> Esip-cf at lists.esipfed.org
>> http://www.lists.esipfed.org/mailman/listinfo/esip-cf
>>
>>
>> End of Esip-cf Digest, Vol 5, Issue 1
>> *************************************
>
>
>
>
> --
> ==== Ted Habermann ===========================
> Enterprise Data Systems Group Leader
> NOAA, National Geophysical Data Center
> V: 303.497.6472 F: 303.497.6513
> "The people who are crazy enough to think
> they can change the world, are the ones
> who’ll do it" Apple
> ==== Ted.Habermann at noaa.gov ==================
>
>
> _______________________________________________
> Esip-cf mailing list
> Esip-cf at lists.esipfed.org
> http://www.lists.esipfed.org/mailman/listinfo/esip-cf
>
More information about the Esip-cf
mailing list