[Esip-documentation] on documenting instrument and platform metadata
Jim Biard
jbiard at cicsnc.org
Fri Mar 20 11:39:24 EDT 2015
Bob,
You are quite right about use metadata. I wasn't paying close attention,
and put Use in the acronym I created just because it made acronym
contain the word CUP. I guess it would be ACP - Attribute Conventions
for Provenance. Or perhaps PAC - Provenance Attribute Conventions. Or
CAP - Conventions for Attributes for Provenance.
Anyway, I wasn't trying to supplant CF.
Grace and peace,
Jim
On 3/20/15 11:16 AM, Bob Simons - NOAA Federal via Esip-documentation wrote:
> Isn't "use" metadata CF's realm? I know Ethan Davis has a proposal in
> to EarthCube to expand the use of CF to other earth science domains.
> Shouldn't we leave "use" metadata to CF?
>
> Is there a real need for Yet Another Standard? If you want new "use"
> metadata attributes, wouldn't it be best to work with the CF group to
> add them to CF?
>
> The nice thing about standards is that there are so many of them to
> choose from.
> -- Andrew S. Tanenbaum
>
>
> On Thu, Mar 19, 2015 at 9:58 AM, John Graybeal via Esip-documentation
> <esip-documentation at lists.esipfed.org
> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>
> I have always seen the Venn diagram of these (provenance, use,
> discovery) as overlapping. Interesting to pursue further.
>
> If there is a call today I'm afraid i will have to miss it.
>
> John
>
> On Mar 19, 2015, at 9:00 AM, Ge Peng - NOAA Affiliate via
> Esip-documentation <esip-documentation at lists.esipfed.org
> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>
>> Actually, I think that Jim’s suggestion of developing ACUP - the
>> Attribute Conventions for Usage and Provenance is a great idea.
>> Hopefully, it will be picked up by someone and developed by the
>> group in not so distant future.
>>
>>
>> --- Peng
>>
>>
>> On Thu, Mar 19, 2015 at 11:25 AM, Jim Biard via
>> Esip-documentation <esip-documentation at lists.esipfed.org
>> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>>
>> Hi.
>>
>> I think that Nan's last comment is well worth considering in
>> relation to ACDD. Some metadata are focused more on
>> provenance, and some are focused on discovery. The primary
>> purpose of ACDD is to capture discovery metadata, not
>> provenance metadata. That doesn't mean that provenance
>> metadata is unimportant - it can, in fact, be critical to
>> proper use, but that doesn't mean that ACDD needs to address it.
>>
>> Perhaps someone should start developing ACUP - the Attribute
>> Conventions for Usage and Provenance. (grin)
>>
>> Grace and peace,
>>
>> Jim
>>
>>
>> On 3/17/15 11:30 AM, Nan Galbraith via Esip-documentation wrote:
>>> Hi Aaron -
>>>
>>> Having been involved in the oceansites specification on
>>> this, I agree that
>>> no one solution always works. I adopted (and try to follow,
>>> and promote)
>>> the NODC templates as closely as possible. Their development
>>> of the ancillary
>>> variable 'instrument' was brilliant, in my opinion.
>>>
>>> My datasets range from met files where all the data is from
>>> a single
>>> logger but each of 6 or 8 variables might need its sensor
>>> described at
>>> the variable level, to depth merged temperature files where
>>> there's only
>>> one data variable but 30 instruments (and 4 or 5 different
>>> models). In the met
>>> files, variable attributes would work, but I use
>>> 'instrument' ancillary variables
>>> so I can link data variables recorded by a single sensor
>>> module (e.g. Inst_HRH
>>> is linked to data variables HRH and ATMP).
>>>
>>> The depth merged files show the real beauty of the
>>> instrument variable, though;
>>> since it can have a depth dimension, you can describe the
>>> different instruments
>>> deployed at different depths. There's no good way to do
>>> that using attributes, as
>>> far as I've seen - lists break down, especially when data is
>>> sub-sampled i.e. in thredds.
>>>
>>> And, although much of this isn't yet at a stage where
>>> software can use it, any and all
>>> information that you have about the sensors and platform
>>> should still be recorded
>>> wherever it seems most appropriate to you, since it might
>>> otherwise be lost. Re-formatting
>>> this kind of info (if and when a useful standard emerges)
>>> will be much less effort
>>> than trying to track down something that was not recorded
>>> because there didn't
>>> seem like a standard way to encode it.
>>>
>>> If ACDD can't find and understand instrument variables yet,
>>> maybe that's something
>>> that could be looked at in another rev. Or, maybe
>>> instruments aren't a common
>>> discovery concept - I really don't know.
>>>
>>> Cheers -
>>> Nan
>>>
>>>
>>> On 3/16/15 6:41 PM, Aaron Sweeney via Esip-documentation wrote:
>>>> Thanks very much for the guidance, John. I think I may
>>>> stick with the NODC or OceanSites platform and instrument
>>>> variables to define the particular instances.
>>>>
>>>> That being said, for my two use cases, I think I can still
>>>> capture the type of platform and instrument in the ACDD
>>>> global attributes. This amounts to:
>>>>
>>>> :platform = "DART"
>>>> :platform_vocabulary = "NASA/GCMD Platform Keywords.
>>>> Version 8.0"
>>>> :instrument = "BOTTOM PRESSURE GAUGES"
>>>> :instrument_vocabulary = "NASA/GCMD Instrument
>>>> Keywords. Version 8.0"
>>>>
>>>> and
>>>>
>>>> :platform = "COASTAL STATIONS"
>>>> :platform_vocabulary = "NASA/GCMD Platform Keywords.
>>>> Version 8.0"
>>>> :instrument = "TIDE GAUGES"
>>>> :instrument_vocabulary = "NASA/GCMD Instrument
>>>> Keywords. Version 8.0"
>>>>
>>>> Cordially,
>>>> Aaron
>>>>
>>>> On 03/16/2015 02:43 PM, John Graybeal wrote:
>>>>> What great questions! The following thoughts are my own,
>>>>> and are unrelated to my previous ACDD activities.
>>>>>
>>>>> Clearly some data files will have data from multiple
>>>>> platforms and instruments; whereas others will get all
>>>>> their variables from a single platform/instrument; and
>>>>> some will have data so heavily processed as to be barely
>>>>> traceable to the source platforms/instruments. Each of the
>>>>> specifications is oriented toward somewhat different
>>>>> scenarios, and none of them are yet great at representing
>>>>> all the scenarios.
>>>>>
>>>>> Let's consider ACDD first. Note that this specification
>>>>> provides _recommendations_ for attributes, not
>>>>> _requirements_; and the platform and instrument attributes
>>>>> are only 'Suggested'. These give a broad-brush
>>>>> categorization of the platform(s) or instrument(s) that
>>>>> collected a data set; if it seems reasonable to specify
>>>>> that information for the whole data set, use these
>>>>> attributes to do so. (If it isn't useful/applicable, then
>>>>> don't worry about these attributes.) The most precise way
>>>>> to give that info is to use a controlled vocabulary (see
>>>>> https://marinemetadata.org/references?filter0=**ALL**&filter1%5B%5D=129&filter1%5B%5D=141
>>>>> <https://marinemetadata.org/references?filter0=**ALL**&filter1[]=129&filter1[]=141> for
>>>>> a partial list of vocabularies), and typically I'd expect
>>>>> a type vocabulary here (with e.g. "CTD", "Mooring") not a
>>>>> specific instance.
>>>>>
>>>>> If you *are* following ACDD, I don't recommend putting the
>>>>> names of variables containing information on platforms and
>>>>> instruments in the global attributes 'platform' and
>>>>> 'instrument'. Without ACDD describing that as a
>>>>> possibility, it's highly unlikely any software will know
>>>>> how to treat that, and some people will get confused too.
>>>>> The existence of the variables will be pretty
>>>>> self-explanatory, so just leave out those ACDD attributes.
>>>>> (I see this conflicts with NODC and OceanSITES
>>>>> recommendations, to my surprise; perhaps others will offer
>>>>> another view here.)
>>>>>
>>>>> Whether to put the information in a global instrument
>>>>> variable, or attributes of each data variable, or both,
>>>>> seems to me a matter of circumstances and taste. Let's
>>>>> stipulate the best solution is human-recognizable,
>>>>> computer-parseable, extensible, and unambiguous. Thus,
>>>>> global attributes are inadequate. Assuming a sophisticated
>>>>> data system 5 years from now would look in both variables
>>>>> and data attributes, then the model of 'one variable per
>>>>> instrument or platform', and association from the data
>>>>> variables to those variables or platforms seems best.
>>>>>
>>>>> But given that the examples in each of the different
>>>>> profiles still seems inconsistent, likely you should
>>>>> follow the guidance most close to your work. If you aren't
>>>>> particularly close to any of them, I'd do it a bit
>>>>> differently still. But I'll hold that thought unless you
>>>>> want to extend the discussion in that direction.
>>>>>
>>>>> John
>>>>>
>>>>>
>>>>> On Mar 16, 2015, at 11:56, Aaron Sweeney via
>>>>> Esip-documentation <esip-documentation at lists.esipfed.org
>>>>> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>>>>>
>>>>>> Hi, folks,
>>>>>>
>>>>>> I’m struggling with how to capture platform and
>>>>>> instrument information in the appropriate places in
>>>>>> netCDF files. The NODC templates suggest creating
>>>>>> separate variables for platforms and instruments, with
>>>>>> multiple variable attributes (make, model, serial number,
>>>>>> precision, accuracy, etc.), and associating these
>>>>>> platform and instrument variables with physical variables
>>>>>> via an “ancillary_variable” variable attribute attached
>>>>>> to the physical variable. Along these lines, SeaDataNet
>>>>>> suggests including variable attributes for instrument
>>>>>> (sdn_instrument_urn and sdn_instrument_name), and
>>>>>> OceanSites suggests either the use of a separate
>>>>>> instrument variable (with make, model, serial number
>>>>>> attributes) or capturing make, model and serial number as
>>>>>> variable attributes contained directly within the
>>>>>> relevant physical variable. But ACDD-1.3 only has single
>>>>>> /global/ attributes for platform and instrument (and
>>>>>> vocabularies). The NODC templates suggest placing the
>>>>>> names of the variables containing information on
>>>>>> platforms and instruments in the global attributes:
>>>>>> instrument and platform. I like NODC’s and OceanSites’
>>>>>> “associative” approach (i.e. “this instrument goes with
>>>>>> this physical variable”), but the use of variable names
>>>>>> in global attributes conflicts with ACDD-1.3.
>>>>>>
>>>>>> It seems that in order to comply with ACDD-1.3, I
>>>>>> need to capture instruments and platforms at a global
>>>>>> level (dropping their association with physical
>>>>>> measurement variables), but in order to comply with
>>>>>> OceanSites and others, I need to capture instrument and
>>>>>> platform information in separate variables and associate
>>>>>> these with physical measurement variables via an
>>>>>> ancillary_variable attribute.
>>>>>>
>>>>>> In order to comply with both, in the interest of
>>>>>> enabling interoperability, I seem to need to repeat
>>>>>> instrument and platform metadata in two different places.
>>>>>>
>>>>>> Any thoughts or guidance?
>>>>>>
>>>>>> References:
>>>>>> NODC netCDF timeseries template:
>>>>>> http://www.nodc.noaa.gov/data/formats/netcdf/v1.1/timeSeriesOrthogonal.cdl
>>>>>> OceanSITES Data Format Reference Manual (See Appendix 2
>>>>>> for sensor variables):
>>>>>> http://www.oceansites.org/docs/oceansites_data_format_reference_manual.pdf
>>>>>> SeaDataNet Data Transports Manual (See Section 4.2.2
>>>>>> Co-ordinate Variables):
>>>>>> http://www.seadatanet.org/content/download/16251/106283/file/SDN2_D85_WP8_Datafile_formats.pdf
>>>>>>
>>>>>> Thanks,
>>>>>> Aaron
>>>
>>> --
>>> *******************************************************
>>> * Nan Galbraith Information Systems Specialist *
>>> * Upper Ocean Processes Group Mail Stop 29 *
>>> * Woods Hole Oceanographic Institution *
>>> * Woods Hole, MA 02543(508) 289-2444 <tel:%28508%29%20289-2444> *
>>> *******************************************************
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Esip-documentation mailing list
>>> Esip-documentation at lists.esipfed.org <mailto:Esip-documentation at lists.esipfed.org>
>>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>>
>> --
>> <hjdfgbcg.png> <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC
>> <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
>> o: +1 828 271 4900 <tel:%2B1%20828%20271%204900>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Esip-documentation mailing list
>> Esip-documentation at lists.esipfed.org
>> <mailto:Esip-documentation at lists.esipfed.org>
>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>>
>>
>>
>>
>> --
>> *Ge Peng, Ph.D*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC
>> <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
>> 151 Patton Ave, Asheville, NC 28801
>> ge.peng at noaa.gov <mailto:ge.peng at noaa.gov>
>> o: +1 828 257 3009 <tel:%2B1%20828%20257%203009>
>> f: +1 828 257 3002 <tel:%2B1%20828%20257%203002>
>>
>> Following CICS-NC on Facebook <http://www.facebook.com/cicsnc>
>>
>>
>>
>>
>> _______________________________________________
>> Esip-documentation mailing list
>> Esip-documentation at lists.esipfed.org
>> <mailto:Esip-documentation at lists.esipfed.org>
>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>
> _______________________________________________
> Esip-documentation mailing list
> Esip-documentation at lists.esipfed.org
> <mailto:Esip-documentation at lists.esipfed.org>
> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>
>
>
>
> --
> Sincerely,
>
> Bob Simons
> IT Specialist
> Environmental Research Division
> NOAA Southwest Fisheries Science Center
> 99 Pacific St., Suite 255A (New!)
> Monterey, CA 93940 (New!)
> Phone: (831)333-9878 (New!)
> Fax: (831)648-8440
> Email: bob.simons at noaa.gov <mailto:bob.simons at noaa.gov>
>
> The contents of this message are mine personally and
> do not necessarily reflect any position of the
> Government or the National Oceanic and Atmospheric Administration.
> <>< <>< <>< <>< <>< <>< <>< <>< <><
>
>
>
> _______________________________________________
> Esip-documentation mailing list
> Esip-documentation at lists.esipfed.org
> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
151 Patton Ave, Asheville, NC 28801
e: jbiard at cicsnc.org
o: +1 828 271 4900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.deltaforce.net/pipermail/esip-documentation/attachments/20150320/ef5a38c8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iibeffdg.png
Type: image/png
Size: 11847 bytes
Desc: not available
URL: <http://lists.deltaforce.net/pipermail/esip-documentation/attachments/20150320/ef5a38c8/attachment-0001.png>
More information about the Esip-documentation
mailing list