Data Discovery Science Competition

 

Come Find the Data You Need!

Grow your skills and your data resources in the Data Discovery Science Competition at DataDiscoveryStudio.org

Data Discovery Studio is built to help scientists find data. It hosts records for over 1.6 million datasets and resources from over 40 repositories, and continues to grow. Individuals, classes, and sci-tech teams can compete in two categories:

(1) Build a shareable Collection of datasets from existing resources, and add links to other datasets and resources in other repositories. We show you how!

(2) Modify existing Jupyter Notebooks or create your own to explore a data type of interest. Guidance provided!

Find details at bit.ly/ddscompete or jump in and start exploring at DataDiscoveryStudio.org – enter by May 1, 2019.

Please share broadly! We built this NSF-funded EarthCube gateway for you, want it to be useful, and eagerly welcome your feedback.

_____

Note to coral reef scientists: the site already includes over 16,600 records of coral reef datasets and resources – enough to begin a select collection of your own to share with collaborators, and point to more.

More Details:

Both competition categories contribute to science. (1) Collections will gather and tag resources, including datasets, with a focus you choose for research or community use. (2) Jupyter Notebooks will help develop workflows for datasets with a common theme or structure. Both areas will help address real science questions, share results and resources with others, and will help improve the gateway for future users. The competition is suitable for individual, team, or class entry.

DDStudio’s strongest holdings in datasets and resources are in earth, ocean, atmospheric, hydrologic, geologic, planetary, ecological, and other geoscience domains, broadly defined. Try out a favorite search term at datadiscoverystudio.org – if DDStudio doesn’t have what you need, you can contribute a new link, or point us to repositories to add. DDStudio goes beyond search: some datasets can be explored right on site using example Jupyter Notebooks (e.g., sensor data) – or develop and contribute your own Notebooks.

You can’t win if you don’t enter – show up and represent your domain! bit.ly/ddscompete

Advertisement

CRESCYNT Data Science for Coral Reefs – Workshop Materials

Thanks to Jan Vincente and Ingrid Knapp for this pair of Data Rescue images.

Written while embedded in our CRESCYNT Data Science for Coral Reefs workshops. Amazingly, everyone who participated in workshop 1 – Data Science for Coral Reefs: Data Rescue – learned even more than they thought they would. We’ve had wonderful NCEAS trainers, spectacular participants with amazing datasets, and a lot of hard work over 4 days (March 7-10, 2018).

UPDATE: Here is the Data Rescue workshop agenda we used, with links to all of the training slides.

In the second intensive workshop – Data Science for Coral Reefs: Data Integration and Team Science – people will be introduced to R Studio and GitHub if they have not used them before, and then we will work on exploring techniques for integrating disparate datasets. We’ll start with a pair of datasets at a time, and efforts may involve extracting data from one dataset based on observations from another; upscaling, downscaling, resampling, or summarizing to make intervals and scales mesh – exactly the kind of process that coral reef researchers have said is a recurring challenge in asking bigger science questions.

UPDATE: Here is the Data Integration and Team Science workshop agenda we used, with links to all of those training slides and exercises.

Each workshop group is writing a paper to summarize and share lessons learned, so please stay tuned for those!

We experimented with an unusual process for these workshops: two days of training followed by two days of workathon. We’re liking it! Tell us what you think about these topics and training materials. What other workshop outputs would you like to see?

>>>Go to the blog Masterpost or the CRESCYNT website or NSF EarthCube.<<<

 

On Preserving Coral Reef Imagery and Related Data – by James W Porter

James Porter’s coral photo monitoring project in Discovery Bay

In preparation for an upcoming Data Science for Coral Reefs: Data Rescue workshop, Dr. James W. Porter of the University of Georgia spoke eloquently about his own efforts to preserve historic coral reef imagery captured in Discovery Bay, Jamaica, from as early as 1976. It’s a story from the trenches with a senior scientist’s perspective, outlining the effort and steps needed to accomplish preservation of critical data, in this case characterizing a healthy reef over 40 years ago.

Enjoy this insightful 26-min audio description, recorded on 2018-01-04.

 

Transcript from 2018-01-04 (lightly edited):

This is Dr. Jim Porter from the University of Georgia. I’m talking about the preservation of a data set that is at least 42 years old now and started with a photographic record that I began making in Discovery Bay, Jamaica on the north coast of Jamaica in 1976. I always believed that the information that photographs would reveal would be important specifically because I had tried other techniques of line transecting and those were very ephemeral. They were hard to relocate in exactly the same place. And in addition to that they only captured a line’s worth of data. And yet coral reefs are three dimensional and have a great deal of material on them not well captured in the linear transect. So those data were… I was very consistent about photographing from 1976 to 1986.

But eventually funding ran out and I began focusing on physiological studies. But toward the end of my career I realized that I was sitting on a gold mine. So, the first thing that’s important when considering a dataset and whether it should be preserved or not is the individual’s belief in the material. Now it’s not always necessary for the material to be your own for you to believe in it. For instance, I’m working on Tom Goreau, Sr.’s collection which I have here at the University of Georgia. I neither made it nor in any way contributed to its preservation but I’ve realized that it’s extremely important and therefore I’m going to be spending a lot of time on it. But in both cases, the photographic record from Jamaica, as well as the coral collection itself – those two activities have in common my belief in the importance of the material.

The reason that the belief in the material is so important is that the effort required to capture and preserve it is high, and you’ve got to have a belief in the material in order to take the steps to assure the QA/QC of the data you’re preserving, as well as the many hours required to put it into digital format. And believing in the material then should take another step, which is a very self-effacing review of whether you believe the material to be of real significance to others. There’s nothing wrong with memorabilia. We all keep scrapbooks and photographs that we like – things relating to friends and family, and times that made us who we are as scientists and people. However, the kind of data preservation that we’re talking about here goes beyond that – could have 50 or 100 years’ worth of utility.

Those kinds of data really do require them to be of some kind of value, and the value could either be global, regional, or possibly even local. Many local studies can be of importance in a variety of ways: the specialness of the environment, or the possibility that people will come back to that same special environment in the future. The other thing that then is number two on the list – first is belief in the material – second is you’ve got to understand that the context in which you place your data is much more important to assure its survival and utility than the specificity of the data. Numbers for their own sake are numbers. Numbers in the service of science become science. It is the context in which you place your data that will assure its future utility and preservation.

CRESCYNT Data Science For Coral Reefs Workshop 2 – Data Integration and Team Science

We’re extremely pleased to be able to offer two workshops in March 2018 at NCEAS. The second is CRESCYNT Data Science for Coral Reefs Workshop 2: Data Modeling, Data Integration and Team Science. Apply here.

When: March 12-15, 2018
Where: NCEAS, Santa Barbara, CA

Workshop description:

This workshop is recommended for early to mid-career and senior scientists with interest in applying technical skills to collaborative research questions and committed to subsequently sharing what they learn. Participants will learn how to structure and combine heterogeneous data sets relevant to coral reef scientists in a collaborative way. Topics covered on days 1 and 2 of the workshop will cover reproducible workflows using R/RStudio and RMarkdown, collaborative coding with GitHub, strategies for team research, data modeling and data wrangling, and advanced data integration and visualization tools. Participants will also spend 2 days working in small teams to integrate various coral reef datasets to practice the skills learned and develop workflows for data tidying and integration.

The workshop is limited to 20 participants. We encourage you to apply via this form. Workshop costs will be covered with support from NSF EarthCube – CRESCYNT RCN. We anticipate widely sharing workshop outcomes, including workflows and recommendations. Anticipate some significant pre-workshop prep effort.

Related posts: Learning to Love R More and R Resources for Visualization

UPDATE: HERE IS THE AGENDA FOR THE WORKSHOP, WITH TRAINING LINKS.

>>>Go to the blog Masterpost or the CRESCYNT website or NSF EarthCube.<<<

CRESCYNT Data Science for Coral Reefs Workshop 1 – Data Rescue

We’re extremely pleased to be able to offer two workshops in March 2018 at NCEAS. The first is CRESCYNT Data Science for Coral Reefs Workshop 1: Data Rescue. Apply here.

When: March 7-10, 2018
Where: NCEAS, Santa Barbara, California, USA

Workshop description:

Recommended for senior scientists with rich “dark” data on coral reefs that needs to be harvested and made accessible in an open repository. Students or staff working with senior scientists are also encouraged to apply. Topics covered on days 1 and 2 of the workshop will cover the basic principles of data archiving and data repositories, including Darwin Core and EML metadata formats, how to write good metadata, how to archive data on the KNB data repository and elsewhere, data preservation workflow and best practices, and how to improve data discoverability and reusability. Additionally, participants will spend approximately 2 days working in pairs to archive their own data using these principles, so applying with a team member from your research group is highly recommended.

The workshop is limited to 20 participants. We encourage you to apply via this form. Workshop costs will be covered with support from NSF EarthCube – CRESCYNT RCN. Participants will publish data during the workshop process, and we anticipate widely sharing workshop outcomes, including workflows and recommendations. Because coral reef science embodies a wide range of data types (spreadsheets, images, videos, field notes, large ‘omics text files, etc.), anticipate some significant pre-workshop prep effort.

Related post: CRESCYNT Toolbox – Estate Planning for Your Data

UPDATE: HERE IS THE AGENDA FOR THE WORKSHOP, WITH TRAINING LINKS.

>>>Go to the blog Masterpost or the CRESCYNT website or NSF EarthCube.<<<

CRESCYNT Toolbox – Discovery of Online Datasets

Data discovery at cinergi.sdsc.edu

Announcing recent progress for data discovery in support of coral reef research!

Take advantage of this valuable community resource: a data discovery search engine with a special nose for locating coral reef research data sources: cinergi.sdsc.edu.

A major way CRESCYNT has made progress is by serving as a collective coral reef use case for EarthCube groups that are building great new software tools. One of those is a project called CINERGI. It registers resources – especially online repositories and individual online datasets, plus documents and software tools – and then enriches the descriptors to make the resources more searchable. The datasets themselves stay in place: a record of the dataset’s location and description are registered and augmented for better find and filter. Registered datasets and other resources, of course, keep whatever access and use license their authors have given them.

CINERGI already has over a million data sources registered, and over 11,000 of these are specifically coral reef datasets and data repositories. The interface now also features a geoportal to support spatial search options.

The CINERGI search tool is now able to incorporate ANY online resources you wish, so if you don’t find your favorite resources or want to connect your own publications, data, data products, software, code, and other resources, please contribute. If it’s a coral-related resource, be sure to include the word “coral” somewhere in your title or description so it can be retrieved that way later as well. (Great retrieval starts with great metadata!)

To add new resources: Go to cinergi.sdsc.edu, and click on CONTRIBUTE. Fill in ESPECIALLY the first fields – title, description, and URL – then as much of the rest as you can.

Try it out!

Thanks to EarthCube, the CINERGI Data Discovery Hub, and the great crew at the San Diego Supercomputer Center and partners for making this valuable tool possible for coral reef research and other geoscience communities. Here are slides and a video to learn more.

 

>>>Go to NSF EarthCube or the CRESCYNT website or the blog Masterpost.<<<

CRESCYNT Toolbox – Data Repositories – Estate Planning for your Data

“Hypotheses come and go but data remain.”    – Ramon y Cajal

Taking care of our data for the long term is not just good practice, allowing us to share our data, defend our work, reassess conclusions, collaborate with colleagues, and examine broader scales of space and time – it’s also estate planning for our data, and a primary way of communicating with future scientists and managers.

Here are some great options for long-term data storage, highlighting repositories friendly to coral reef science.

First, there are some important repository networks useful for coral reef data – these can unify standards and offer collective search portals: we like DataONE (members here) and bioCaddie (members here).

KNB – the Knowledge Network for Biocomplexity offers open and private data uploads; ecological orientation. DataONE network.

NOAA CoRIS: Coral Reef Information System – often free to use and can accept coral reef related data beyond NOAA’s own data; contact them first.

BCO-DMO – Biological and Chemical Oceanography Data Management Office – if you have an NSF grant that requires data storage here, you’re fortunate. Good data management guidelines and metadata templates, excellent support staff. Now a DataONE member.

Dataverse – supported by Harvard endowments. There are multiple organizational dataverses – the Harvard Dataverse is free to use. bioCaddie member.

Zenodo – free to use, supported by the European Commission (this is a small slice of CERN’s enormous repository for the Large Hadron Collider). Assigns dois. We invite you to include the “Coral Reef” community when you upload. bioCaddie member.

NCBI – the National Center for Biotechnology Information is very broadly accepted for ‘omics data of all types. A bioCaddie member.

DataCite – not a repository, but if you upload a dataset at a repository that does not assign its own doi’s, you can get one at DataCite and include it when publishing your datasets.

We’ve not listed more costly repositories such as Dryad (focused on journal requirements) or repositories restricted to institutions. What about other storage options such as GitHub, Amazon Web Services, websites? Those have important uses, but are not curated repositories with long-term funding streams, so are not the best data legacy options.

Most of these repositories allow either private (closed) or public (open) access, or later conversion to open access. Some have API’s for automated access within workflows. These are repositories we really like for storing and accessing coral reef work. Share your favorite long-term data repository – or experiences with any of the repositories listed here – in the comments.

CRESCYNT Toolbox – What do we need on the workbench?

Science is a team sport. Collaborations allow us to ask more ambitious science questions, but also intensify the need to connect disparate datasets across scales of time and space. Solving data interoperability challenges requires technological solutions not yet in place, so we’re taking the initiative to review potential solutions.

We really want to include YOUR favorite tools and highest priority needs in this review, so please share those – we’ll report back.

A platform from EarthCube is some time and distance away, but we have a chance to start assembling tools already at hand and in use for coral reef research workflows and do some testing. The process also helps us ground the ideal in the practical.

What are  some criteria for a great infrastructure platform?

Ideally, solutions are: 1) modular, so when an improved tool is available it can be incorporated without restructuring the system; 2) free or low cost, so solutions are sustainable for most research labs; and 3) open source, allowing continued development from multiple disciplines and directions. However, we also want to start where people are, with the tools we’re already using – many of these are less than ideal but we make them work. That’s our starting place, and we want to hear about all of your tools.

It is tempting to set up a workbench for the challenge of analysis alone, but in a coral reef research lab we immediately crash into the realities of group data collection, field and lab work, physical specimens, and intersecting projects. All of these characteristics create additional layers of challenge. In the long run, infrastructure should help capture data and metadata generation at the source, and ease tracking, analysis, and replicability.

Good infrastructure solves more problems than it creates in compliance, skill demand, and management. An effective system helps graduate students and postdocs develop robust skills in managing data into the future, with guidelines that work for people, labs, and collaborators. Interoperability challenges must be solved for datasets that range from remote sensing to ecological surveys to bioinformatics work. Data cleaning, analysis, visualization, and mapping must be supported in flexible ways to clearly communicate research insights.

We have an opportunity to construct a preliminary array of tools and workflows on a cyberinfrastructure workbench for coral reef research, so if this is going to be a broadly useful start we need to know what you already like using and what more you wish you had. Dream big to start with, and along the way we’ll acknowledge the distinction between perfect and good-enough solutions.

Please complete this survey, or comment below. Thank you so much!

CRESCYNT Toolbox – One Search through Multiple Data Repositories

“Search One and Done”… don’t you wish there was a KAYAK search tool for datasets?

Currently, the closest thing to a single search interface that pulls from a broad collection of heterogeneous data repositories at diverse scales is DataONE (http://dataone.org). Available DataONE repositories (“member nodes”) that host a substantial number of datasets about and relevant to coral reef work include:

• LTER Long Term Ecological Research network (e.g., Moorea Coral Reef LTER),
• NOAA NCEI Nat’l Centers for Environmental Information (incl. historic sea surface temperature data, some coral reef monitoring data),
• Dryad (wide ranging publication-associated datasets),
• KNB Knowledge Network for Biocomplexity (including coral reef studies),
• and many more (https://www.dataone.org/current-member-nodes)

 

You can get your own data into DataONE through KNB or ONEShare member nodes if other repositories are not a good fit.

DataONE continues to be actively developed and improved, and to add new member nodes. It currently provides access to over 630,000 metadata and datasets. Don’t miss DataONE’s Investigator Toolkit (https://www.dataone.org/investigator-toolkit).

Please check it out and let us know what you think!

 

 

>>>Go to the blog Masterpost or the CRESCYNT website or NSF EarthCube.<<<

WELCOME to CRESCYNT – the Coral Reef Science and Cyberinfrastructure Network

The Coral Reef Science & Cyberinfrastructure Network (CRESCYNT) is a multi-tiered and multidisciplinary network of coral reef researchers, ocean scientists, cyberinfrastructure specialists, and computer scientists, and we invite you to join us. 

As an EarthCube Research Coordination Network, our goals are to foster a dynamic, diverse, durable, and creative community; to collectively consider and develop standards and resources for open data, research documentation, and data interoperability while making best use of work already accomplished by others; and to offer input to those groups within EarthCube who will ultimately create the data architecture for all of EarthCube. Along the way CRESCYNT expects to collect and share community resources and tools, and to offer training opportunities in topics prioritized by our members through widely accessible formats such as webinars and their recordings. We will also work to nurture unforeseen collaborative opportunities that emerge from our integrated collective work.

Because the coral reef community has exceptionally diverse data structures and analysis requirements needed to forward integrative science, it is an exemplar for cyberinfrastructure-enabled advances to other geosciences communities. The CRESCYNT network is working to match the data sources, data structures, and analysis needs of the coral reef community with current advances in data science, visualization, and image processing from multiple disciplines to advance coral reef research and meet the increasing challenges of conservation. The network has begun to assemble to coordinate, plan, and prioritize cyberinfrastructure needs within the coral reef community.

The structure of CRESCYNT is a network of networks, currently including 18 disciplinary nodes and 7 technological nodes, where each network node represents an area of coral reef science (disciplinary nodes: e.g., microbial diversity, symbiosis regulation, disease, physiology & fitness, reef ecology, fish & fisheries, conservation & management, biogeochemistry, oceanography, paleontology, geology) or an area of computer science or technical practice (technological nodes: e.g., visualization, geospatial analysis & mapping, image analysis, legacy & dark data, database management). These nodes may expand, coalesce, or divide to meet the needs and interests of the subdisciplinary communities, while maintaining connections to CRESCYNT through node coordinators and ongoing network activities. We invite you to become a member of CRESCYNT, join one or more nodes that would advance your own work, collaborate on shared resources and tools for the coral reef community, and ensure that the data architecture and cyberinfrastructure of EarthCube will meet the needs of the coral reef community, and that broader data interoperability within EarthCube will benefit both coral reefs and our ability to answer complex questions.

PLEASE VISIT OUR WEBSITE at http://crescynt.org to enroll in CRESCYNT, join a node, work on tasks, discuss data and research priorities, and help determine the future shape of cyberinfrastructure for supporting coral reef research and other geoscience work. This collaborative work is supported by the NSF EarthCube initiative.  Dr. Ruth D. Gates, Director of the Hawaii Institute of Marine Biology, University of Hawaii, is the Principal Investigator of the CRESCYNT project. The CRESCYNT blog is written by Dr. Ouida Meier, the project’s program manager (crescyntrcn@gmail.com).

This material is based upon work supported by the National Science Foundation under Grant Number 1440342. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

>>>Go to the blog Masterpost or the CRESCYNT website or NSF EarthCube.<<<