Batch processes for Wikidata .. importing from ORCiD and Crosreff - a more comprehensive trick

Every week a process runs that produces a list of all the papers for all the scientists known to Wikidata that have an ORCiD identifier. The papers are known by a DOI and typically all scientific papers at Wikidata have a DOI. ORCiD-Scraper uses this list for interested users to upload the information of these papers to Wikidata using the "QuickStatements" tool. One paper at a time for one author at a time.

What if.. what if all new papers of all authors known to ORCiD are added? The challenge will be not to introduce duplicate papers or duplicate authors.. So lets agree on two things, we only introduce authors who have an ORCiD identifier and for now we only introduce papers who have a DOI and at least one author who has an ORCiD identifier.

The trick is to cycle through all authors known to Wikidata. For instance a thousand at a time. All new papers have their DOI entered in a relational database where a DOI can exist only once. All these papers may include multiple authors, they enter the same relational database where an author is unique. When all papers for the first thousand authors and associated authors in the database, we can first add all missing authors to Wikidata and add the Wikidata identifiers in the relational database. We then add the missing papers. It is likely that no duplicates are introduced but there will be duplicates for authors where Wikidata does not know about the ORCiD identifier.

We cycle through all our known authors and we can rerun each week.

We can do this but why.. 

One reason is an English Wikipedia template named {{Scholia}} it may be used on scholars and subjects. Unlike Wikipedia the information it presents will always be up to date. There are more reasons but this is a great starter.

Thanks,

      GerardM

3 million is a lot of Wikidata edits

Today there is a Wikicite conference. Today I reached 3 million Wikidata edits. It was about a scientist by the name of Budd, he won an award.

Wikicite would be a big thing when it has room for growth. It should contain all the references used in Wikpedia, it could contain all the awards known to Wikipedia and all its recipients. It would be great when all publications referencing Wikipedia sources would be known; it would be a clue to how much Wikipedia readers are missing out on.

When we know about all the publications of the CDC scientists at Wikidata, thanks to a Scholia presentation we would know what America has developed so far. It is a lot, we could celebrate it.

This is the first blogpost for me for 2025. I still dream big but what I achieve is one edit at a time.

Thanks,

       GerardM

Making Wikicite a success by putting the Wikipedia editor first

Wikicite as a project has come to an end after a five year run. Much has been achieved (read the article). One great follow up idea is to harvest all the references of all the other Wikipedias and have all this data together so that we can analyse our quality even better.

All the data, all the Wikicite activities have been valued; people have been thanked, accomplishments announced and it is now left open how to move on. The best way to move forward is to bring a public to the data and add value. 

Every Wikipedia article has its references and we put Wikipedia editors and readers first when we show all the known references and its relations with older and newer publications. One button that provides the latest information will create buy in, makes it more interesting. It links to papers, to authors and requested updates are not part of a serial but of a targeted process.

A similar service we can provide for the authors of scholarly works. They will find in Wikidata what we know about them, identifiers of VIAF, ORCiD, Google Scholar, ResearchGate, Twitter even Scopus. They can improve the information on their publications they can add identifiers, references to other publications, co-authors, subjects and see it improve the associated Scholia representations. What we can do for them is regularly harvest information and update Wikidata with the new or altered information.

When we truly build relationships, it is no longer essential that everything should be on Wikimedia servers. Why have a same project at the Internet Archive and at the Wikimedia Foundation? Why not share the work load. When we put our Wikipedia editors first, we provide them with the best information on literature, publications we have on offer. Most references are in the WaybackMachine anyway, we already rely on this so why not collaborate and share both the effort and the cost?

When we truly care about sharing the sum of all knowledge, it is in the eating that we find the proof, not in the dogma.

Thanks, GerardM

The promise of things to come that is in @Wikicite

Wikicite is for me the biggest disappointment of all the Wikimedia projects. A disappointment not because it is not worthwhile, not because it did not bring us insights about what happens in English Wikipedia but because all the papers, conferences and data did not result in a transformation into active user stories. Closest comes Scholia where it identifies Wikipedia articles where a scholarly paper is used as a reference. 

Wikicite conferences have come and gone. They brought the best and brightest minds together discussing all kinds of ideas, all kinds of research. Once the papers were presented, the assertions discussed and analysed, when the conference was done they went home. From what I can observe in Wikimedia land, not much has changed.

Wikimedia Foundation has a real good friend in the Internet Archive. It already does much of what Wikicite could do. It archives everything that is used as a reference. When a reference goes dark, it links the reference to the backup. In FatCat it has information on scholarly papers and many of the papers, in Open Library it has information on books and many of the books. Many bots originating from the IA run on many Wikimedia projects. 

When it is up to me, I would have Wikicite as a joint project of the Internet Archive and the Wikimedia Foundation. A joint project will be based on the existing reality that is in the Internet Archive. Wikicite and Wikimedia projects bring it additional data, a public and additional user stories. Funding from the Wikimedia Foundation enables the development that such a synergy brings.

Data for all the citation in English Wikipedia linked to scholarly papers was available. We know in Wikidata, in FatCat, in Orcid, in Scopus how to disambiguate authors. When all this data gets integrated, a user story mentioned in a previous blog post is not fancy and easily becomes best practice. Thanks to collaboration with the Internet Archive there is less duplication of effort and the sum of the shared knowledge we hold is so much bigger; we can provide an even better service to our public. 

Science Is Shaped by Wikipedia: Evidence From a Randomized Control Trial

A scholarly paper has it that a paper cited in Wikipedia gets a bigger exposure in other scholarly papers. That is in itself quite huge, it speaks of the impact that Wikipedia has in scholarly circles.

When the cited papers trigger citations, it follows that the new papers represent much of the scientific development of the subject of the Wikipedia article. That makes it obvious that in order to keep Wikipedia articles up to date, it is relevant to know the papers that cite these scholarly articles.

To help editors, Scholia already provides assistance. There is a Scholia for each paper; this one is for the paper I referred to earlier and, there is a random Scholia for a subject identified in Wikidata. Just think what a Scholia specific for a Wikipedia article could be and do.

Maintaining information is a drag and there are many bots that are used to append and amend the information that we have based on what we have in a Wikipedia article. First, there is the Internet Archive bot that save guards information on websites by making a copy for the Wayback Machine. For many of the references we know a DOI and it is easy enough to ensure that we know the associated paper in Wikidata. It is easy enough to make a process out of it for a single paper.

What it takes is a different take on processes; not the traditional serial process with serial results but a  focused process that supports a user story. 

In the initial phase of a review of a Wikipedia article, the Wikipedian starts a process that updates all the citations to the existing Wikipedia references. Once the process is done, the citations are known, these papers have been fleshed out with open information and the Wikipedia will know the latest science for the subject reviewed.

Much of the functionality exists, when the user story is supported, it is not only Wikipedians and scholars but also our public that is invited to share in the sum of what we know.

Thanks, GerardM

Wikicite from the ground up: "Trophic rewilding"

In nature conservation, trophic rewilding and trophic cascades are important topics. When an animal like the howler monkey is no longer around, it no longer distributes the seeds of trees. The likely effect is that in time plants are no longer part of the ecosystem. Reintroducing a howler monkey restores the relation; it is considered an example of trophic rewilding.

At Wikipedia there is no article about trophic rewilding. As someone famously said, references are the most important part of a Wikipedia article, let's start with finding references.

There is a longstanding process of importing data about scholarly papers, all kinds of scholarly papers. Some of them have "trophic rewilding" in their title. Trophic rewilding was not known as a subject so it was easy enough to look for "trophic rewilding" and add it as a subject. Slowly but surely the Scholia representation evolves. More papers means more authors and more authors known to have collaborated on multiple publications. More citations are found for these papers and by inference they have a relation to the subject.

The initial set of data is already good enough to get a grasp of the subject but when you want more, you can look for missing data using Scholia, information like missing authors. The author disambiguator aids in finding papers for the missing author. With such iterations, the Scholia for trophic rewilding becomes more complete.

Another avenue to improve the coverage of a subject is by adding "cites work" in Wikidata for a paper like this one. Not all cited works are known to Wikidata but the effect can be impressive. NB The citations are often found in a PDF  and not in the article..

Slowly but surely all the scholarly references to be used for a new article are available, you can use a template in the article to link to the (evolving) Scholia. The best bit is you can add this template in an existing Wikipedia article as well providing a scholarly rabbit hole for interested readers.

Thanks, GerardM

Wikicite, but from the bottom up

Wikicite is one of the most active projects in Wikidata. Its purpose is to "develop open citations and linked bibliographic data to serve free knowledge". A lot of work has been done over the years, there is only one issue; what purpose does it serve.

One of the visible parts of Wikicite are the many Scholia presentations for information. Papers, authors, organisations, subjects even combinations. There is a template that enables the inclusion of Scholia information on a Wikipedia article like here.

One objective of Wikicite is to become the repository of all references of Wikipedia articles. This is where progress is possible enabling people like myself to combine the two and make it easier for Wikipedia editors to find even more sources.. I spend a lot of time adding the subject "trophic cascade" to scholarly articles that include the phrase "trophic cascade" in its title. In addition I attributed the papers of many a scholar as well. This is reflected in the Scholia for trophic cascade. Many of the papers in the references part of the English article are these same papers.

Referenced articles may be specific to multiple subjects and, may be part of the references of multiple articles. When we know all the papers used as references in a Wikipedia article in Wikidata, we can make the information in a Scholia even more useful. 

The information of existing papers with authors and citations can be enriched. For references we can add new papers., the subject of the Wikipedia article can be marked as a "main subject" for the paper as well. We weave a mighty web in this way. Our quality will be improved by flagging retracted papers and we can flag articles for an update when new information becomes available as well.

What we do does not have to be complete. That is not the way, that is not the Wiki way. When we start with what we have, we will find that it is already really useful.

Thanks, GerardM

There is much more to read; introducing the "one page wonder"

Given that our aim is to share in the "sum of all knowledge", realistically we will not have it all at our disposal to share. It is also fairly likely that we will not know about all subjects.

When you google for a given subject, it is as likely as not that you will drown in too much data, too many false friends or find nothing at all when there is nothing to find in "your" language.

Increasingly, what we know about in the Wikiverse is linked to a Wikidata item. Pictures may depict a subject, articles may be written about a subject and all of them refer to a Wikidata item that may have labels in any language. Items that may even have links to references.

When we are to find for Wikipedia readers more to read, we need a mechanism, a place where we can link a subject to external resources. Resources like the Internet Archive, "your" library. the papers we know in WikiCite but to the free versions of these papers. The page will show the label in "your" language and  a picture. It links to all the pictures depicting the subject as well.

Putting the "one page wonder" in production is easy. It is all on one page and is fully internationalised. The localisation is done at translatewiki.net and when people want to make it useful in "their" language, they will add the missing labels for the Wikidata items.

With the "one page wonder" in place it becomes interesting:

• Is "your" local library known to us and do we get your permission to find it for you. How do we supply "your" library with a search string?
• The Internet Archive's wayback machine may have content in "your" language but can you navigate its English only user interface. 
• What other organisations do we want to partner with to provide you with more to read
• Will we be able to show local pictures, a Dutch cow looks different from an Indian cow..
• What other issues will there be..
• Oh and yes, we can include the Reasonator, queries and what have you.. we just have to think about what to show.

Thanks,

       GerardM

Professor @steve_hanke and reading what is #FAIR

Professor Hanke is on Twitter. He has his five Wikipedia articles and his info on Wikidata is well developed. With a scholar of his eminence, you would expect a lot of known publications as well. However, never mind the 153 English Wikipedia references, never mind the links to 13 external authorities, finding his work is not easy nor obvious.

The problem with Wikipedia references, it is a hodgepodge of links about him and links to his works. His VIAF registration may bring you some of his works but it will not tell you where his books are cited. Mr Hanke does not have an ORCID identifier and consequently it is not easy to include his data on Wikidata.

This is not about Mr Hanke; in certain fields of science people do not have an ORCID identifier or are not open about their publications. When you are interested in a specific subject or a specific scientist, it helps when the information is FAIR.

So what is missing; there is this database with all Wikipedia references, it needs to be included in Wikidata as soon as possible. It may require a fair deal of social manoeuvring to include all Wikipedia references to Wikidata. But the benefits; the benefits will be huge. Given that Wikipedia references are backed up by the Internet Archive, this will extend for these links in Wikidata as well. It makes them FINDABLE and ACCESSIBLE. At Wikidata, this data becomes INTEROPERABLE and REUSABLE (FAIR).

So my 2019 wish for the Wikimedia Foundation is to become FAIR in what it says and what it does.
Thanks,
      GerardM

Bringing more #science to @Wikidata

Slowly but surely more scientific papers and their authors find their way into Wikidata. Particularly when scientists have staked their claims in ORCiD, adding is easy and obvious.

It is easy because in ORCiD every author, paper, organisation et al have their own unique identifiers. So when you add a paper, all authors who claimed to be author are already linked.

Earlier today, I added papers and co-authors for Jaume Piera. As a consequence Laura Recasens was added today and as you can see in the illustration of her co-authors, several new authors popped up as a consequence.

To do this I use a combination of tools. Reasonator is my preferred tool to display data; for scientists it tells me if he or she is known to be an author. When there are, Scholia presents the scholarly author information. Of particular relevance to me is the co-author presentation. For co-authors shown in white, no gender is given in Wikidata and when the name is an initial and a surname, I will look up the ORCiD information to find a full name. Typically that is how people are known in ORCiD.

I use the SourceMD tool for two purposes; "creating and amending papers for authors" and to "add metadata from ORCiD authors to Wikidata". It is processed in a batch job, I run one job for up to 15 authors at a time and it takes forever to run.

Other people run other jobs, a particular hat tip to Daniel Mietchen who makes sure that recent publications find their way into Wikidata and finds many other reasons to improve on what we have. All this would not be possible without the many tools by Magnus and for Scholia I do thank Finn Årup Nielsen thanks to this evolving presentation, science as a process comes alive.

There is more to do; the Wikipedia citation are in a separate database and much of its data may be found in Wikidata.. Who will merge them. Publications do cite other publications, it is a field I am not really interested in.. They are added so there must be a tool.

When you are interested in a particular scientist, a particular paper.. Just use the tools and slowly but surely we all make Wikidata a great tool to represent science fact.
Thanks,
      GerardM

#WikiCite: Edith Abbott; THE REAL JAIL PROBLEM

According to #Wikipedia, Edith Abbott was a pioneer in the profession of social work with an educational background in economics.She published, books and scholarly articles, and "The Real Jail Problem" was singled out in the article for special mention.

There is a link in the article, and I am happy to report that thanks to the Internet Archive, it and some 500.000 more scholarly articles may be found there. They are part of the early JSTOR papers and they are now freely available. It is however not the publication by Mrs Abbott, it is by a Robert H. Gault.

Having scientific papers available is wonderful. It is important and it is unlikely they will get lost. However, there is a difference between being lost and being findable. Mr Gault is notable, there was no mention of him in Wikidata, one of his publications was.. Mr Gault is findable when you look him up in VIAF.

However, the article you want to read, a pamphlet according to Mr Gault is where? A book with the same subject can be found at Open Library. The subject of the book is as potent as it was in the day. The arguments are not dissimilar.

Arguably, when you want to cite your sources, they first have to be knowable, then findable and finally readable. Wikidata is making more and more of a difference. When we collaborate with the Internet Archive, we can make those JSTOR papers findable as well. Slowly but surely all these sources becomes findable. Its authors become notable and we will free us from the curse of only reading the latest research.
Thanks,
       GerardM

#Wikicite: #Research? Eat your own dogfood!

Another piece of Wiki research has been published and you would not know when you look for it in the Wikiverse. The people who are big in Wiki research run this project called Wikicite. It has multiple aspects; publications and there authors are entered in Wikidata. Blogs are written how wonderful the various aspects are of what is on offer in the growing quality and quantity of data and how well an author may be presented in Scholia.

This is all well and good and indeed there is plenty to cheer about like the documentation about the Zika fever that is included but when a subject that is key to the Wikimedia researchers is not as well represented it will never be good enough.

There is a reason why using your own is so relevant; it shows you where your model fails you. WAll kinds of everything like conference speakers are included, many more female scientists are represented as a result of the indomitable Jess Wade but when new research by Wikimedians, professional Wikimedians, are not included, the effort is not sincere; the people that do go to conferences do not learn from the daily practice and that makes Wikicite stale and mostly academic.
Thanks,
     GerardM

#Wikimedia - #diversity and #inclusion requires #trust

All the Wikimedia projects have their culture. Each project has its own culture and there is this overall stated ambition that diversity and the inclusion that is needed to make it work is alive and well.

We have our diversity conferences and the best result is how the "gender gap" is approached. It gets a lot of attention and the positive effects are noticeable. There is however more to diversity and some of the beliefs we hold so dear prevent the inclusion from those that are at the outside looking in.

One of the Wikimedia traditions is that we do not trust; trust is in the citations, the sources. We do not trust each other, why should we? When for diversities sake, people who receive the "Harriët Freezerring" are added, it is accepted because there is a Wikipedia article that mentions them.. But when people are added because they are "artists from the African diaspora" there is a problem. There are no articles yet and the point of adding the artists first is because Wikidata enables managing projects in multiple languages.

There are many people who are targetted for attention in Wikipedia editathons. There have been editathons in the past so there is an established track record for the Black Lunch Table. That did not bring trust, the trust needed to accept that the BLT will manage the people on the list. The trust that bare boned items will get sufficient statements eventually.

The problem with trust is that when it is not given, it can not be assumed for other, similar situations either. The trust that retractions from scientific papers will be included so that we know what Wikipedia articles are inherently wrong. Retractions are absent at this time and while I trust the people involved in the inclusion of citations, why trust at all when equally worthy causes are not trusted? Why include all these scientific papers without similar quality control?
Thanks,
       GerardM

#Wikipedia - #Retraction exposing big issues in #science

When a scientific paper is published, it is read and cited by other scientists to further on science. It is read and cited by Wikimedians to write articles and share the sum of all knowledge. The Wikicite project provides better tooling for using these papers as a source in Wikipedia articles, it is one of the more relevant developments in combatting fake news in Wikipedia.

However.. there is an issue with a substantial number of papers; they were retracted. There are all kinds of reasons possible but the bottom line is; they are not to be used as a source in Wikipedia because its findings are false.

The challenge: what papers are retracted, how are retractions and the reasons for retractions modelled and how will we find these papers in the Wikipedia sources. Knowing retractions and acting on them will be a fine art; one publisher in South Africa for instance was pressed to retract a book exposing the president. There will be so many issues exposed once retractions become part of the Wikipedia work flow. Failing to do so will be the worst we can do. We will not be sharing the sum of all knowledge, we will be sharing the sum of what we are told.
Thanks,
       GerardM