Showing posts sorted by relevance for query atom. Sort by date Show all posts
Showing posts sorted by relevance for query atom. Sort by date Show all posts

Friday, April 1, 2011

A World of Numismatic Data at Your Fingertips

The JISC Common Repository Interfaces Group (CRIG) has a saying: "The coolest thing to do with your data will be thought of by someone else." The American Numismatic Society's MANTIS interface enables users to search, sort, and browse the collection in a way that is relevant to their own research questions. Users can view query results in the form of HTML pages or OpenLayers maps that draw on KML layers. These are probably the two most common avenues for interacting with the ANS's data, but we have developed several APIs that expose the data in the form of linked data and KML.

Currently, we offer an Atom feed and a KML API. An RDF feed is currently in the works and will be released in the coming weeks. These APIs are based on queries to the Solr search index and can be adjusted based on the user's demands.

Atom: http://numismatics.org/search/feed/?q=*:*
Object KML: http://numismatics.org/search/query.kml?q=*:*
Mint KML: http://numismatics.org/search/mints.kml?q=*:*

The feeds are formulated based on the Lucene query syntax. q=*:* passes a query for all results from all fields (i.e., the entire collection of objects). For example, http://numismatics.org/search/feed/?q=department_facet:"Greek" returns the Atom feed for all coins in the Greek department with 100 results per page. This is particularly useful to those who want to subscribe to the portion of the collection that is relevant to their own interests. The practicality of this feed extends beyond casual subscriptions to updates to the database. The Atom XML contains a reference to the next page in the feed. Each object entry includes a pointer to the HTML record of the coin, as well as pointers to other representations of the data, including, importantly, the source Encoded Archival Description XML file. This allows anyone to scrape the entire ANS collection. With more than 600,000 objects, the Atom feeds creates the potential for interesting quantitative analysis.


The KML APIs works similarly--feed it a Solr query and get results back that you can look at in Google Earth or a similar geographic framework. There are two APIs. The object API (query.kml) returns at most 500 results for the query, with points representing individual objects. The mint API returns a point for each mint the query illuminates. This is the foundation for the OpenLayers map function in MANTIS. There is no limit on the number of mints returned by this API. Providing access to thousands of coins (or the entire collection!) in a single KML file would probably bring down the server.

Refer to the Lucene query syntax linked above on appropriate text to pass to the q parameter. There are a wide variety of fields, detailed below. The field name typically corresponds with numismatic ontology or internal ANS database fields.

Text Fields
  • color_text
  • unitid_text (accession number)
  • legend_text
  • type_text
  • persname_text (any personal name: artist, portrait, issuer, etc.)
  • deity_text
  • geogname_text (geographical name, including mint, locality, region, and findspot)
  • fulltext (keyword search anywhere in the record)

Sortable Numbers
  • axis_sint
  • weight_sint (weight)
  • era_sint (century)
  • year_sint
  • dob_sint (date on object)

String Fields
  • color_display
  • dimensions_display
  • imagesponsor_display
  • unitid_display
  • unitdate_display (date of coin, in textual terms. e.g. 16 BC - AD 3)
  • unittitle_display
  • obv_leg_display (obverse legend)
  • obv_type_display (obverse type)
  • rev_leg_display (reverse legend)
  • rev_type_display (reverse type)
  • reference_display
  • prevcoll_display (previous collection)
  • sernum_display (serial number)
  • category_display
  • timestamp
  • artist_facet
  • authority_facetusefulness
  • category_facet
  • city_facet
  • decoration_facet
  • deity_facet
  • degree_facet
  • denomination_facet
  • department_facet
  • engraver_facet
  • era_facet
  • famname_facet (Dynasty)
  • findspot_facet
  • format_facet (object type)
  • issuer_facet
  • persname_facet (Person)
  • locality_facet
  • manufacture_facet
  • magistrate_facet
  • maker_facet
  • material_facet
  • moneyer_facet
  • mint_facet
  • region_facet
  • script_facet
  • series_facet
  • state_facet
  • subject_facet
  • subjectPerson_facet
  • subjectIssuer_facet
  • subjectEvent_facet
  • subjectPlace_facet

Images
The 'imagesavailable' field can return objects with images if the value is set to 'true'

Geographic
  • mint_geo (geographic coordinates)
  • mint_kml (nomisma.org mint KML URI)

A few notes. String fields (including mint_geo and mint_kml) require exact matches (enclosed by double quotes if the string is providing access to thousands of coins (or the entire collection!) in a single KML file would probably bring down the server.more than one word), but can accommodate Solr wildcards (* and ?).

Have at it!

Examples
http://numismatics.org/search/feed/?q=department_facet:"Islamic" AND weight_sint:[4 TO 5] - All Islamic coins that weigh between 4 and 5 grams

http://numismatics.org/search/feed/?q=type_text:temple AND imagesavailable:true AND department_facet:"Greek" - Greek coins depicting temples that have been photographed

Thursday, September 8, 2011

Leveraging nomisma to Deliver Controlled Vocabulary Services

With the launch of the nomisma.org Atom feed just a few days ago, I went to work on developing an XForms component which leverages the service to deliver controlled vocabulary terms for ancient mints within the administrative backend of Numishare. Numishare's interface for creating and editing coins already included a connector to geonames.org's APIs for querying modern place names, which I have detailed on the EADitor blog.

The nomisma service operates in similar fashion. A user selects a "nomisma" radio button (versus the other options to query geonames or local controlled vocabulary stored in the Solr index), inputs a search term into a text box, clicks the search button which executes an xforms:submission to query the Atom feed (which populates an xforms:instance), and selects the appropriate option from the select list box, which is formed by each atom:entry returned by the query. An example is pictured below:


The nomisma URI is stored as an attribute in the mint element in the NUDS-XML record so that, upon indexing the record into the search index, a lookup is performed on the RDFa of the nomisma id to extract geographic coordinates, enabling mapping out of the box of ancient mints (in addition to modern mints that are supplied through the geonames service). As noted earlier, findspots are also mappable.


Additionally, if the nomisma RDFa includes a reference to the associated Pleiades place, the Pleiades URI is also stored in the search index, enabling querying of Numishare by that URI.

Code for XForms XBL component

Tuesday, September 6, 2011

Nomisma.org launches Atom feed

Last week I began development on an Atom feed for the numismatic concepts described on nomisma.org. The feed will be updated nightly with the latest changes to the concepts, so you can be sure to keep abreast of new mints, denominations, hoards, etc.! Like the Atom and KML services on MANTIS, the feed is derived from Solr search results, and is thus queryable with the Lucene syntax. Most importantly, this feed will form the basis for numismatic controlled vocabulary in the backend of Numishare and will ultimately facilitate internationalization of coin collections that use the software by indexing coins based on unique identifiers for typological attributes (e.g., denomination, material) that are independent of the language in which the institution's collection is described.

There is a limited number of fields, so I will list them below.

id: unique identifier of the record
typeof: category of record. values currently include-- hoard, typology_entry, mint, skos:Concept, authority, romanemperor, collection, material, numismaticTerm, term, and denomination
prefLabel: preferred label of record
altLabel: alternate label of record, likely to include other languages
definition: small blurb describing the object
pos: GML-compliant geographic coordinates
burial_start: integer referring to terminus post quem (of a hoard)
burial_end: integer referring to terminus ante quem (of a hoard)
fulltext: text anywhere in record

Examples
http://nomisma.org/feed/?q=typeof:mint+attica - Mints in Attica
http://nomisma.org/feed/?q=burial_start:\-240 - terminus post quem of 240 B.C.

The feed contains links to the next and last pages so that it is possible to easily page through it. The page number is appended after feed/ in the URL, like so:

http://nomisma.org/feed/2/?q=*:*

Many of the entries in the feed also include geographic coordinates, and thus Google Maps is capable of rendering it in map form.

I will follow this post up with another about the feed's interaction with the XForms backend in Numishare, and how this will contribute to name authority.

Thursday, October 2, 2014

Numishare Migrated from Cocoon 2.1.11 to Orbeon

Over the past few weeks, I have been working on migrating Numishare's public user interface from Cocoon 2.1.11 to Orbeon. While Cocoon has been capable of delivering most of the functionality integral to Numishare, it is time to move on. Cocoon has been part of Numishare's application stack since before Numishare really even became a reusable framework; it dates back to the development of the University of Virginia Art Museum Numismatic Collection website back in 2007. Since this date, Cocoon has seen the release of 2.2 (dramatically different than 2.1.11), and has been stuck in a 3.0 alpha for more than three years. In fact, the most recent release of any version of Cocoon is 2.1.12. The application has served the cultural heritage community well for years, but let's admit it, Cocoon is dead. It has been dead for years, and I've been keeping Numishare going with duct tape and bubble gum for far too long. In fact, you have to change one of Cocoon 2.1.11's out-of-the-box settings to even support native UTF-8, resulting in garbled responses from Solr.

Although Cocoon has run its course, XML technology is still quite powerful, and has gotten a shot in the arm with the new XQuery/XPath 3.0 and XForms 2.0 specs, which will support JSON processing, among other new features. Orbeon's XML Pipeline Language (XPL) is quite robust, and Orbeon has been powering the user interface for all of my other projects for quite a few years (xEAC [see: http://numismatics.org/authorities/], EADitor [see: http://numismatics.org/archives/], the new [yet-to-be-released] Nomisma.org, and Kerameikos.org).

All of the functionality built in Cocoon's pipelines have been ported into Orbeon's XPL, which is now a better MVC framework. Furthermore, XPL and Orbeon processors enable a variety of modern HTTP features that are employed for the semantic web. All pages are HTML5, and record pages now include RDFa to embed Nomisma properties.

It is now much easier to deliver linked data in RDF/Turtle and JSON-LD. Numishare supports content negotiation to receive individual record metadata in NUDS/XML, RDF/XML, Turtle, JSON-LD (all three in Nomisma's model and ontology), and KML in the appropriate content-type; search results are available in Atom by requesting 'application/atom+xml' or raw Solr XML by requesting 'application/xml' in the Accept header. I have not included it yet, but the Pelagios linked data will also be made available in TTL and JSON-LD in addition to RDF/XML. The multilingual interface will be more dynamic, as we can make use of the browser language passed through the request header rather than relying on the 'lang' URL parameter. Finally, I have introduced an 'apis' page (linked from the navigation menu), which lists available APIs (just one now, to aggregate NUDS records into a single model) and relevant documentation. There are a number of features that have yet to be added, but Orbeon provides a path for implementation that Cocoon simply does not.

Most of the projects on our test server have already been migrated into the new version of Numishare. Once I have ported the stylistic themes from OCRE, AoD, CHRR, and Mantis, I will work on pushing the new Orbeon-based Numishare into production.

Friday, July 12, 2013

Updates to Nomisma Atom Feed

With the launch of the new Nomisma.org, the Atom feed pagination format will change slightly and be more consistent with the format introduced by Numishare applications.

In the older Nomisma, pagination was treated in the URL as follows:

http://nomisma.org/feed/2/?q=*:*

In the new version, the query more closely resembles Lucene queries:

http://nomisma.org/feed/?q=*:*&start=100

Additionally, burial_start and burial_end have been removed, in addition to the geographic search.  I recommend using the SPARQL interface for these sorts of sophisticated queries.

The new feed incorporates opensearch and includes alternate links to the RDF serialization for each entry.  See the original blog post for further documentation.

Friday, November 6, 2015

Using XForms to transform Google Spreadsheets into RDF

With XML Amsterdam happening and reading about a lot of the XForms action happening at the conference, it has occurred to me that I haven't written any blog posts in the last few months about technical advances in Nomisma or other projects (like the Digital Library application, which features an XForms interface for MODS that accepts a PDF upload, and sends the PDF into Solr for full-text indexing, as well as dynamically generates EPUB files from TEI). The ETDPub application will get a full technical write-up in a journal eventually, probably code4lib.

VoID CRUD

On Nomisma, I worked on a few new features in the backend that have greatly reduced my workload. First, I finally implemented a system that makes use of the VoID metadata RDF for data dumps that are contributed into the Nomisma SPARQL endpoint to facilitate the aggregation of coins for OCRE, CRRO, and our other large projects. The VoID RDF is validated to ensure it contains the required title, description, license, etc., and if valid, the data dump is ingested into the SPARQL endpoint. The dump can be refreshed with a click of a button, or a dataset can be removed from the triplestore entirely by passing in a SPARQL/Update query:

PREFIX nmo:    <http://nomisma.org/ontology#>
PREFIX void:    <http://rdfs.org/ns/void#>
PREFIX dcterms:    <http://purl.org/dc/terms/>
DELETE {?s ?p ?o} WHERE {
{ ?object void:inDataset <DATASET> ;
  nmo:hasObverse ?s . ?s ?p ?o }
UNION { ?object void:inDataset <DATASET> ;
  nmo:hasReverse ?s . ?s ?p ?o }
UNION { ?object void:inDataset <DATASET> ;
  dcterms:tableOfContents ?s . ?s ?p ?o }
UNION { ?s void:inDataset <DATASET> . ?s ?p ?o}


This is a fairly simple workflow, but it isn't yet complete in that it only accommodates RDF/XML at the moment (need to expand for Turtle and JSON-LD), nor does it validate the data dumps. This saves me a lot of time in that I can simply click a button in the user interface to re-ingest a dump when new matches are made between coins in that dump and new types published in OCRE--or simply refresh the OCRE dump when we publish new types. Before, I used to have to shut down the triplestore for a few minutes, delete the data directory, and then manually upload each RDF dump into the triplestore via command line.

Google Spreadsheets to RDF


The other really significant advancement has reduced my workload significantly with respect to batch publication of new concepts (as RDF) into Nomisma. I would occasionally receive spreadsheets of data to upload into Nomisma, which required me to author a PHP script to transform CSV into RDF, and there were invariably validation problems in the original data.

I spent 1-2 weeks developing an XForms application that could read a Google Spreadsheet (published as an Atom feed) in order to validate the data and import as RDF. First, one begins with a spreadsheet like this.

The user will be presented with an interface like the one below:


The user may map the spreadsheet headings to allowable RDF properties. There are some basic requirements--that there be a Nomisma ID that conforms to xs:anyURI, that there be one English preferred label, one English definition, that there may be no duplicate languages for preferred labels or definitions. There must be both a latitude and a longitude if uploading mint IDs. That sort of thing. The full list of allowable properties and more specific instructions are at https://github.com/nomisma/framework/wiki/Import-Update-IDs.

After selecting a valid data mapping (XForms bindings), the user may proceed to the next screen, which then validates each row in the spreadsheet to ensure the data values conform to other bindings. For example, there cannot be blank values for English preferred labels, and skos:exactMatch, skos:closeMatch, skos:broader, and the like must be URIs that begin with the https?:// regular expression (XPath matches() function). If everything is valid, the XForms engine will transform the Atom XML into the appropriate RDF/XML model, save to the filesystem (for versioning in Github), post to the SPARQL endpoint, and then transform the RDF/XML into an XML document for indexing into the Solr search index.

There's a neat additional feature that executes when there's a skos:closeMatch or skos:exactMatch with a wikipedia or dbpedia URL. An XForms submission executes that queries Wikidata based on the article title to extract titles in other languages (essentially facilitating multilingual interfaces by mapping alternate languages into skos:prefLabel in RDF) as well as matching concepts in other vocabulary systems, like VIAF, the Getty AAT/TGN/ULAN, Geonames, etc. In the end of this process, we can some pretty sophisticated RDF that can link people to URIs in other systems and model their relationship to a political entity or dynasty with the org ontology, e.g., http://nomisma.org/id/muhammad_ahmad_al-mahdi.

We have created nearly 1,000 new Nomisma concepts this summer through this new spreadsheet import mechanism--a great investment in two weeks' of labor to free me from having to write data processing scripts and pushing the responsibility of creating and updating IDs to the numismatic subject specialists.

This import mechanism is open source of course: https://github.com/nomisma/framework/blob/master/xforms/import.xhtml

Wednesday, August 14, 2019

Recommendations for numismatic spreadsheet standardization

Over the years, we have considerably refined the way in which we organize our spreadsheets for processing into NUDS XML files and upload into the Numishare platform. Our workflow started with Online Coins of the Roman Empire, where numerous interns worked over the course of four years (the final three funded by the NEH) to produce dozens of spreadsheets (typically one per emperor) encompassing more than 40,000 types.

Many of the primary typological categories, such as denomination, mint, and authority, contained Nomisma URIs, and textual categories, e.g. legend and type description, were columns of free text. These spreadsheets (Excel files) were exported into CSV and processed through a PHP script that I wrote to transform each row into a NUDS document, and then this batch of files would be uploaded with the eXist-db XML database client into the appropriate Numishare collection. After this, I would manually edit the code in the Admin panel in Numishare to index the most relevant batch of RIC IDs into Solr for the public-facing browse and search interfaces (so as not to reindex an entire collection of 40,000 types when a new or updated spreadsheet might only contains several hundred items).

With the publication of PELLA in 2015, we implemented a key->pair stylesheet that enabled us to connect obverse and reverse type description codes to each unique description, with columns for English, French, and German translations. The OCRE PHP script was modified to accommodate this new model. Subsequent type corpora have been published for Ptolemaic and Seleucid coinage, each with a slight variation of yet another PHP script. Furthermore, with partners in the Netherlands, Switzerland, England, and Italy deploying their own Numishare collections for type corpora and/or collections of physical specimens, the wide range of slightly different spreadsheet models require an ever-diverse set of scripts that need to be manually maintained. It has long been a goal of mine to implement a standardized spreadsheet import into Numishare itself, modeled on the XForms-based validation and transformation of Google Sheet's Atom XML API implemented several years ago in Nomisma.org.


Mapping Google Sheets columns to NUDS elements


Finally, after about a month of development and testing, a Google Sheets-based spreadsheet import is functional in Numishare. It is primarily focused on type corpora at the moment, as not all of the physical and administrative descriptors have been implemented for mapping spreadsheet column headings of numismatic objects.

Some things remain the same:
  • Typological categories must map to Nomisma URIs
  • References for physical objects can be a coin type URI of some sort, a plain literal, or a combination of a type series and type number separated by a | character. The type series must be a literal or a Nomisma URI for a type series, but I am to enable support for Zenon bibliographic URIs
  • Parent IDs (skos:broader) and deprecation-related IDs (dcterms:replaces or dcterms:isReplacedBy) must be contained in the spreadsheet.
  • A question mark can trail a Nomisma URI to denote uncertainty. This is parsed in the XForms engine to insert the appropriate uncertainty URI into the NUDS XML. 
  • Columns for symbols/monograms located at certain positions on the obverse and reverse can be mapped to the positions listed in the Numishare config.

Structured XML produced from a spreadsheet

Other types of information requirements must be met in order for the spreadsheet to validate, which means that certain data must be explicit and not automatically inserted by a script. For example, each NUDS XML document requires a title. This title was typically generated in the PHP script by some concatenation of a human-readable string with the type number parsed from an ID column. Similarly, all coin types and all physical specimens NOT linked to a coin type URI must have an Object Type URI in the spreadsheet, even if that URI for all objects is nm:coin.

All of this normalization can occur in a pre-processing phase in OpenRefine: automatic generation of titles through regular expressions, reconciliation of typological columns to Nomisma URIs through Nomisma's OpenRefine API, etc.

This new spreadsheet import also requires type descriptions to be present in the typological spreadsheet, which means rethinking the way in which descriptions are connected to the main typology spreadsheet. Instead of a separate stylesheet spreadsheet of key->pair combinations between codes and translations, this stylesheet is incorporated as a second sheet in the typological spreadsheet. It is therefore possible to create a VLOOKUP formula between the unique type description code in the typology sheet and the corresponding column in the description stylesheet (see https://docs.google.com/spreadsheets/d/e/2PACX-1vQoyHYDyh79oJuoW9m2g9BNbnysyVWjl13KQNEyTF5dgXswQwgekXMvIDTAH3onwN35c1P9eXeJAD4w/pubhtml). Therefore, the type descriptions can still be maintained with the ease of making one change to a description in Sheet #2, and the change will immediately propagate into the Atom feed.

VLOOKUP to control type descriptions

I have applied the same logic for concordances. A single concordance sheet can be maintained and propagated across multiple relevant  type corpora.

See for example the Svoronos 1904 corpus of Ptolemaic coinage: https://docs.google.com/spreadsheets/d/e/2PACX-1vSSxfdRUvq_PZOlvt3Od1T1gu29wOSQub6DwqQviq1TMRs2gDCWRA4u0i0cqHaHWchJ9Zt3pq03pc0t/pubhtml

This contains a partial concordance between Svoronos numbers and the types from Catharine Lorber's Coins of the Ptolemaic Empire vol I, part I (gold and silver from Ptolemy I - IV as published in Ptolemaic Coins Online).

By eliminating the intermediary scripting and XML upload/indexing process, scholars will be able to use OpenRefine to prepare their data without much technical intervention and publish their type or specimen data into Numishare without significant IT overhead. This alone will save me quite a lot of time: a month of development up front to save at least the same amount of time per year in redundant scripting and OpenRefine data cleaning.

After a spreadsheet is uploaded, it will be indexed directly into Solr, if the types are active (not deprecated by newer URIs) and the indexing option has been enabled.

Full documentation of the spreadsheet upload is forthcoming.

Thursday, July 18, 2013

Nomisma: Using XForms to Manage and Publish Linked Open Data

One of the main improvements in the newly-redesigned Nomisma web architecture is in the administrative backend, not visible to the public.  The previous iteration of Nomisma was built on top of open source wiki software.   Each id was an XHTML+RDFa fragment in the filesystem, created and edited through the wiki.  There was no validation, and the hand-coding of XHTML fragments occasionally led to human error: invalid XML documents which occasionally broke page loads or RDF distillation.  We needed to move to a more stable and scalable infrastructure.

The XHTML+RDFa fragments remain a part of the new architecture of Nomisma, now maintained in a GitHub repository.  The fragments are now edited in an XForms interface with the Orbeon processor, which enables not only editing of XML, but a variety of REST interactions to get and post data into the Apache Fuseki RDF triplestore and SPARQL endpoint, and post data into the Solr search index, which powers the Atom feed.

While the XForms web forms handle the simplest of XHTML templates, such as those for authorities, mints, regions, etc., it does not yet handle editing of more the more complex data models, such as those for IGCH hoards (like http://nomisma.org/id/igch0200) or coin types (for example, http://nomisma.org/id/rrc-174.1).  However, hoards and coin types are least likely to be manually edited, so the editing interface is most useful for those numismatic concepts which are most likely to be enhanced with additional labels and references to other linked open data identifiers (like VIAF or Pleiades ids).

Validation


One of the main features of XForms is advanced validation.  The @typeof attribute in the XHTML root div is tied to a drop down menu.  The values in this drop down menu are generated dynamically before the form has finished loading (xforms-model-construct-done) directly from a SPARQL query to acquire all of the nm:numismatic_term ids in Nomisma:

PREFIX rdf:      <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dcterms:  <http://purl.org/dc/terms/>
PREFIX nm:       <http://nomisma.org/id/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT ?uri ?label WHERE {
?uri  rdf:type <http://nomisma.org/id/numismatic_term>.
?uri skos:prefLabel ?label .
FILTER (lang(?label) = "en")}
ORDER BY ASC(?label)

A similar query is passed from Orbeon to the endpoint to generate an XForms instance for nm:field_of_numismatics (e.g., Greek Numismatics, Roman Numismatics, etc.).  Languages (xml:lang in the div) are also tied to an instance which contains every ISO language code and label.


XForms bindings and XPath also ensure that other requirements of the XHTML document are met: there must be an English preferred label, the labels cannot be blank, there can be no repetitive languages for preferred labels, latitude and longitude must be decimal values between -180 and 180, and related links must be valid URIs.

One of the new features of this interface is the interaction between XForms and dbpedia.  It is possible to import labels in languages not already in the Nomisma id from dbpedia RDF.  The XForms submission is fairly straightforward:

<xforms:submission id="get-dbpedia-rdf" action="http://dbpedia.org/data/{instance('control-instance')/dbpedia}.rdf" ref="instance('dbpedia')"
                replace="instance" method="get">
                <xforms:message ev:event="xforms-submit-error" level="modal">Failed to get Dbpedia RDF.</xforms:message>
                <xforms:action ev:event="xforms-submit-done" xxforms:iterate="instance('dbpedia')//rdfs:label">
                    <xxforms:variable name="lang" select="@xml:lang"/>
                    <xforms:action if="not(instance('doc')/xhtml:div[@property='skos:prefLabel'][@xml:lang=$lang])">
                        <xforms:insert context="instance('doc')" nodeset="./xhtml:div[@property='skos:prefLabel'][last()]" origin="instance('prefLabel-template')"/>
                        <xforms:setvalue ref="instance('doc')/xhtml:div[@property='skos:prefLabel'][last()]" value="context()"/>
                        <xforms:setvalue ref="instance('doc')/xhtml:div[@property='skos:prefLabel'][last()]/@xml:lang" value="$lang"/>
                    </xforms:action>
                </xforms:action>
</xforms:submission>
Thus it is easily to rapidly and easily incorporate new labels into Nomisma to facilitate multilingual interfaces in other projects which depend on it for data (like OCRE and the UVA collection).

Workflow


Since the ids need to be maintained in GitHub in the long-term, the editing workflow requires the loading and saving of XHTML+RDFa fragments in the filesystem rather than through a REST interface like eXist.

The workflow is as follows:
  • Load existing id from filesystem or create new one
  • Edit the id
  • Save id. When the document is valid, the save button becomes enabled, and clicking the save button initiates several processes:
  1. Serialize the ids to XML and save back to the filesystem
  2. Serialize the XHTML+RDFa into RDF
  3. Using SPARQL/Update, POST the RDF back into the endpoint.  Since using POST adds new triples into the subject (e.g., http://nomisma.org/id/rome) (creating duplicate triples), the subject must first be flushed from the endpoint before the RDF is sent to Fuseki.  Therefore the following SPARQL query must be sent to the endpoint before the newly-edited RDF is inserted (wonky and unintuitive, but necessary with SPARQL/Update):
DELETE {?s ?p ?o} WHERE { <http://nomisma.org/id/rome> ?p ?o . ?s ?p ?o . FILTER (?s = <http://nomisma.org/id/rome>) } 
  • After the RDF is updated in the endpoint, the XHTML+RDFa is serialized into a Solr XML document and posted into the search index (for the Atom feed, although we may implement faceted search/browse eventually). After the doc is sent, a commit is sent to Solr.
  • Finally, a nightly cron job adds new files into the GitHub repo, and then changes are committed and pushed into GitHub.  Another job then runs to generate RDF dumps of the Nomisma data, which are available on the nomisma.org home page.
This is the gist of the editing workflow in the new version of Nomisma.  I plan to improve the XHTML+RDFa editing templates to support a greater degree of complexity in the data model.  Additionally, I aim to create an administrative interface to better manage datasets provided by other institutions.  The endpoint includes not only Nomisma ids, but RDF provided by OCRE, UVA, CHRR, the ANS, and a portion of the Berlin coinage for Augustus.  I want to be able to get VoID RDF files from new data contributors and do consistency checks on RDF dumps before ingesting them into Fuseki.  I also want to be able to delete or update all triples from a single institution.  This functionality will come eventually.  It will become a higher priority once there are more contributors of numismatic data to Nomisma.

All the code discussed above is, of course, open source: https://github.com/ewg118/nomisma/tree/master/xforms

Friday, October 23, 2015

More than 700 Greco-Roman mints updated in Nomisma

Thanks to Ryan Baumann's work of creating a concordance between geographic identifiers in the Pleiades Gazetteer of Ancient Places and the Getty Thesaurus of Geographic Names, Dan Pett of the British Museum was able to build on this work to incorporate these concordances into the Portable Antiquities Scheme database. Dan's Nomisma-Pleiades-TGN concordance R script is on Github.

Dan then emailed the Nomisma listserv with a large CSV document of all mints in the PAS database, with associated Nomisma IDs, Getty, BM, Geonames, dbPedia, Pleiades, etc. I stripped away all of the mints that don't already have Nomisma IDs so that I could upload the CSV into Google Sheets, which then makes it possible to import data from the Atom representation of this spreadsheet into the Nomisma RDF. I expanded all of the concordance ID columns into full URIs for the Nomisma spreadsheet validation process, and then successfully updated 721 Greco-Roman mints to add Getty, BM, Geonames, and dbPedia URIs as skos:closeMatch objects. Further, the spreadsheet import process parsed the dbPedia URIs to perform Wikidata lookup, enabling us to add further concordances extracted from Wikidata--including the Wikidata URI itself, plus GND, BnF, and Freebase identifiers. The Wikidata lookup also adds additional translations as skos:prefLabels in from article titles in other languages.

As a result, we have added more than a dozen new translations for Zeugma and a few additional URIs.

Monday, August 31, 2020

First pass at mapping Nomisma.org to Linked Art JSON-LD

I have made some updates to both Numishare and Nomisma.org's back-ends to facilitate improved and more consistent content negotiation. I have applied a Content-Language header to the response for coin/type pages in Numishare, based on whether the language passed via request parameter (?lang) or content negotiation by the browser settings (Accept-Language) is present within the Numishare translations. 

More importantly, though, is that the Link header has been added into the requests for coin/type URIs in Numishare (as well as the browse page, because a user can request the Atom XML or untransformed Solr XML via content negotiation). The Link headers have also been applied to the URI for any concept or concept scheme in Nomisma.org. 

This follows examples established W3C Content Negotiation by Profile draft.

A `curl -I` of a URI includes a a Link to itself as a text/html resource with rel="canonical", but it will also include a list of other comma separated  rel="alternate" resources found at the same URI, but with an array of acceptable content types (different serializations of RDF, the NUDS/XML document, KML, and/or GeoJSON) and, in some cases, an additional profile URI. 

 

Requesting headers for http://nomisma.org/id/rome

For most serializations of RDF that are available (RDF/XML, Turtle, and JSON-LD), the profile URI is set to http://nomisma.org/ontology#, to reflect the Nomisma data model for coins, types, and SKOS concepts. Another JSON-LD serialization is available in the Linked Art profile, https://linked.art/ns/v1/linked-art.json. It is now possible to request this profile for a Nomisma concept through content negotiation, e.g.:

curl -H "Accept: application/ld+json;profile=\"https://linked.art/ns/v1/linked-art.json\""
    http://nomisma.org/id/rome


Content negotiation was already possible in Numishare, but now the Link headers indicate to a script or robot that additional content types and profiles are available for consumption.

The Nomisma.org Linked Art JSON-LD follows the spec for People and Organizations (including dynasties) and Places. For other types of concepts in Nomisma, they are defined generally as crm:E55_Type, but are not distinguished further by material, denomination, object type, etc. We will implement these classifications as they are documented in the Linked Art specification. Here's a representation of Seleucus I according to this model: https://gist.github.com/ewg118/8ccab36087a42b71ef15ece7b160f09a. This includes the preferred label in English, the English definition, matching URIs, and memberships in the Seleucid Empire and Seleucid Dynasty. A further conneg request of the latter two URIs include further context about those organizations, including the formation and dissolution dates.

In theory, a parser capable of reading Linked Art JSON-LD for a Human Made Object (for example, a coin in our collection) can crawl from the object data into any associated Nomisma.org concept. A parser can request the HEAD for a Nomisma URI, read that a serialization for the Linked Art profile exists, request the data via content negotiation for the content type and profile, and then harvest or display additional contextual information about the entities related to the object.

I got these serializations up and running in about a day, and my next tasks will be to copy and paste these new functions into Kerameikos.org's back-end and xEAC, so that it will be possible to serialize EAC-CPF archival authorities into Linked Art JSON-LD for broader reuse in Linked Open Data systems.

Tuesday, December 9, 2014

Extending Nomisma for spatial queries

I have recently worked on extending the geographic query functionality in the new Nomisma prototype. It took a few days of fiddling around and testing, but it was altogether a painless operation. First it required reading the (excellent) spatial query documentation for Apache Fuseki, which is the SPARQL endpoint that we have been using in Nomisma for nearly two years. I actually had not upgraded Fuseki since launching it into production in the early months of 2013, and so I had to update from 0.2 to 1.1 across the board in order to enable spatial queries--on the Nomisma production server, on the American Numismatic Society production server (Fuseki powers the interlinking of the Society's archival records in Archer), and on my two development machines at home.

Fuseki's configuration can be extended to use Lucene or Solr as the spatial index. The latter made a lot of sense because Solr is already an integral part of the architecture of most of our digital projects. The Solr index in Nomisma powers the basic facet-based search/browse interface and the Atom feed. I found an older email on the Jena list with an example configuration TTL file that I was able to modify.

After successfully configuring the spatial index, I initially learned that I was not able to use my current index for extending Fuseki. That is because only main Nomisma concepts are indexed, and not the geo:SpatialThing objects associated with mints and regions. The browse interface would be confusing if you received results for both nm:rome and nm:rome#this, the geographic feature associated with the concept of the mint of Rome. As a result, I spun up a new index for storing the URIs and coordinates for the geo:SpatialThings only, and this facilitated spatial queries that delivered results that I expected.

This is a dramatic leap forward in the types of geographic queries we can do in Nomisma, as we can get query results within a bounding box or within a certain distance of a point. Once we begin connecting larger numbers of coins with findspots into the system, we'll be able to do heatmaps showing geographic distribution and a variety of other visualizations. If we ingest other forms of GIS data, such as roads, springs, shrines, etc., we can query for hoards and finds within X meters of the geographic feature.

The example below is a query that will get mints within 50 km of Athens (by giving the lat and long of the city).

SELECT * WHERE {
   ?loc spatial:nearby (37.974722 23.7225 50 'km') .
   ?mint geo:location ?loc ;
         skos:prefLabel ?label .
  FILTER langMatches (lang(?label), 'en')
} 


You can test it yourself in the Nomisma prototype.

Tuesday, March 8, 2011

Mapping Part 2: Context

The new American Numismatic Society collection interface will enable users to navigate from records of coins to other objects related by various attributes (issuer, deity, material, etc.) via hypertext links, which redirect the user to the search results page for that particular query. This is one of the features that made the University of Virginia Art Museum Numismatic Collection such a versatile site when it went live in early 2008. Given our ability to map query results in OpenLayers with a Solr-to-KML transformer, we wanted to take the mapping up a notch by visualizing relationships geographically that had only been represented textually in other Numishare sites.

A small map is embedded in a jQuery tab. This map has several layers, each representing facets, that can be turned off and on as the user requires.


Each mint or cluster of mints on the map can be clicked to display a popup that displays the mint name(s) and available facets at that location. The list of facets is represented by a set of checkboxes, and a link within the popup opens a search results page with the selected facets at that mint. The task of displaying checkboxes within the popup is harder than it seems. OpenLayers does not support selecting multiple features on more than one layer at a time, so a handful of Solr queries must be executed upon clicking the feature.

  1. The mint names of the feature are gathered.
  2. For each visible layer, Solr is queried to see if there are any search results for that facet located at that mint (e.g., is there at least 1 result for Nike at Tyre, Sidon, or Berytus?)
  3. If so, return a checkbox via Ajax with some prepopulated value.
  4. When the user clicks on the "View" link, javascript is fired off to assemble the values of the checkboxes into a proper Solr query and opens the search results in a new browser window.

Using the map interface from the object page enables us to formulate far more complex queries than the linking approach does. Once the user arrives at the search results, he or she can easily adjust the query by adding/removing facets or sorting the results by a wide variety of fields. There are currently several alternate data delivery options offered through the linked icons on the top left of the screen: an Atom feed of the query (useful for scholars who may only want to subscribe to Roman Republican coins, for example) and a KML representation of the query that can be loaded by Google Earth. These are just some of many new features the new ANS collection interface will exhibit when it launches within the next few weeks.

Monday, November 19, 2012

New APIs introduced to Numishare

Over the last week or two, I have devoted some time to developing a systematic (and documented) set of APIs for querying and extracting data from Numishare.  These advances have been applied to OCRE already and will be available in the new version of Mantis, due out in the near future.  Moreover, since I have heard one or two requests about actually downloading data en masse from Mantis, I developed a rather simple PHP script which allows the user to do this.  The PHP script has been added to the Numishare codebase in tools/get-ids.php.  The comments of the script contain the basic execution instructions.

Documentation for Search API: http://wiki.numismatics.org/numishare:search

Documentation for Get API: http://wiki.numismatics.org/numishare:get

The Search API documentation details the two response formats (Atom and RSS) and the Lucene query Syntax (since the search results are derived from Solr).  Most importantly, the wiki lists all of the fields which are used for searching and sorting and their data types, as well as several usage examples.

Thursday, June 18, 2020

About 1,300 Seleucid Monograms published to SCO

About 1,300 monograms appearing on the coinage of Seleucus I through Antiochus III (Houghton, Lorber, and Hoover's Seleucid Coins, Part 1) have been published to Seleucid Coins Online (SCO) and linked to thousands of types and subtypes. Much like previous work with PELLA and PCO, the constituent letters have been annotated by Peter van Alfen, although I am sure we will spot some errors or oversights in the process.

Where the SCO monograms differ from the previous projects is that monograms have been organized hierarchically by Oliver Hoover. Unlike in PELLA and PCO, where a monogram usually has one SVG file (but may have more than one to illustrate slight variations in style), the Seleucid monograms have been grouped thematically by letters and general design motif.

SCO monogram 299


For example Monogram 299 generally consists of Μ, Ε, and Υ, and has four sub-monograms linked to the parent through skos:broader in the RDF. These sub-monograms URIs (e.g, http://numismatics.org/sco/symbol/monogram.houghton.299_3), are encoded within the coin type data, which makes it possible to visualize the mints, hoards, and coin types connected at the sub-monogram level as well as the parent level, by extending the underlying SPARQL query all types associated with the parent monogram through each of the children:



  BIND (<http://numismatics.org/sco/symbol/monogram.houghton.299> as ?symbol)
  {
    SELECT ?symbol ?side WHERE {
     {?side nmo:hasControlmark ?symbol}
     UNION {?children skos:broader+ ?symbol .
              ?side nmo:hasControlmark ?children}
    }
  }


Similarly, Seleucid Coins is an extensively hierarchical type corpus. Many symbols and monograms are annotated at the subtype level. While individual subtypes are indexed into Solr in Numishare for OpenRefine reconciliation and the Atom feed (which powers lookups in KENOM), the browse page itself shows only parent types. The underlying code in Numishare was updated so that symbols and monograms annotated within subtypes are indexed into Solr, making it possible to access them from the symbol facets, regardless of whether that particular monogram appears explicitly within the data for the parent type. For example, Monogram 426 appears in SC 42.2, but has been indexed into the data for SC 42 so that it still appears on the browse page for the query, with two other non-hierarchical types in which the monogram appears directly. Coincidentally, Monogram 426 only appears on posthumous Alexanders minted in Carrhae from 310-290 B.C.

Query results for Monogram 426



Some technical advances I would like to implement in these user interfaces in the near future:

  • A histogram showing the dates of production for monograms
  • Plugging the monograms into the existing distribution analyses functions in Numishare and Nomisma (to generate charts showing authorities, mints, etc. that produced monograms)
  • Vary the radius of the mint points on the map by density of production
  • Introduce a draggable timeline into the map so that the geographic distribution can be visualized over time

The links between letters, monograms, types, mints (and other typological attributes), and hoards will open the door to completely new types of query that were not possible before the application of Linked Open Data to numismatics. We are barely scratching the surface of what's possible. I have recently queried a list of concordances between Ptolemaic monograms and mints from the Nomisma.org SPARQL endpoint that I plan to import into a network visualization tool, like Gephi, for experimentation.

Thursday, February 12, 2015

New Nomisma.org framework has been released

After much work, the new Nomisma.org framework has been launched into production. Not only is this a major architecture migration (moving the public UI from Apache Cocoon into Orbeon), but a data migration. We have implemented a new data model for IDs that conforms to the Nomisma ontology that Karsten Tolle has been working on for at least two years. IDs have a more stable system of classes that will improve the predictability of queries. There are presently 16 classes by which IDs are defined--most of them in the nmo: (http://nomisma.org/ontology#) namespace, but we are using the W3C Organziation ontology for expressing roles of people and organizations (under foaf:). The ontology is still evolving, but has a carefully curated set of properties and classes that pertain to numismatics. Certain URIs (like nm:mint), will never again be used simultaneously as classes and properties. In fact, all of the URIs that were used as classes and properties have been made instances of nmo:NumismaticTerm. Instances of mints (like Rome or Athens) are an nmo:Mint, and nmo:hasMint is the property to use for linking a coin or coin type to a mint URI. The ontology and data conform to standards established by the semantic web and computer science communities.

Data dumps from museum collections (like the ANS and Berlin) have been migrated into the Nomisma ontology, as have RDF exports from online type corpora, like OCRE and CRRO. This involved updating Numishare's code to export in the new ontology (see http://nomisma.org/documentation/contribute for example), as well as update SPARQL queries and the XSLT for reading latitude and longitude in the new model for mints. Mints are now reckoned as concepts (carrying the skos:Concept class, as well as nmo:Mint) that may or may not have a spatial feature, linked with geo:location. The object links by the geo:location is a geo:SpatialThing which may either have a latitude and longitude or have geoJSON encapsulated in the osgeo:asGeoJson property. Complex shapes, represented as geoJSON, can be drawn in Nomisma's XForms-based editing interface (powered by Orbeon). geoJSON objects created in OpenLayers in the editing interface are extracted by Javascript and incorporated into the XForms engine.

The new features of this framework are almost two numerous to mention, but here is a synopsis:

  • IDs are available in their native RDF/XML, but also serialized into Turtle and JSON-LD. IDs for regions that may contain complex geoJSON polygons are exported in geoJSON-LD. These serializations are linked from the ID page. Data dumps of all IDs are available in these three serializations as well.
  • Spatial queries are supported by extending Fuseki to interact with a Solr index.
  • Much improved browse interface allows for additional filtering by roles of people/organizations and sorting. These filters can be applied to the Atom feed as well.
  • Content negotiation is supported for IDs, the SPARQL endpoint, and the browse page. See http://nomisma.org/apis#access for more information about interacting with IDs and Solr. The SPARQL endpoint supports text/html, text/csv, text/plain, application/sparql-results+json, and application/sparql-results+xml.
  • RIC and RRC ids have been deprecated by Nomisma, as OCRE and CRRO maintain up-to-date and better quality versions. HTTP 303 See Other redirects are established for any ID that contains a dcterms:isReplacedBy property that links to something else. See http://nomisma.org/id/rrc-44.5. 
  • We have begun documenting the model and example SPARQL queries. The documentation will evolve to become more comprehensive in the coming months.
  • The system in general is far more stable and efficient now that RDF/XML has replaced XHTML documents. It reduces the need for additional transformation when delivering web services. More than 90% of HTTP requests for Nomisma content comes from machines (about 30,000 per day).

Wednesday, July 20, 2011

Another Take on Visualizing Collections

Today we released a new feature to Mantis--charts and graphs with the handy jQuery plugin Visualize by the Filament Group. This converts HTML tables of data generated dynamically from Solr queries into line, bar, area, and pie charts that are rendered with the HTML5 canvas element. This works in any modern browser (Opera, Safari 3+, Google Chrome, and Firefox 3.5+), though I have found that IE8 has some rendering issues.

The new Visualize feature functions as an extension to the search results page. A chart icon appears in the top of the left-hand column, under "Data Options" (alongside Atom, KML, and CSV deliverables). This will direct the user to a new page with various data and chart output options. The user can select to render a graphical tabulation of available Solr facets given the current query passed via parameter from the search results page. For example, one can receive a chart of deities given the query of Roman coins minted in Asia, Asia Minor, Caesarea, Carrhae, or Carthage. The result is pictured below.


The Solr query and all chart options are passed into the page via URL parameters rather than relying on asynchronous javascript interaction. This means that specific charts have URIs and can thus be bookmarked and passed to others for viewing, e.g., http://bit.ly/r3FUDj. Moreover, the charts are printable!