#Wikidata - #algorithm for updating labels

Amir is the #pywikibot guru; he runs dexbot and it is the only bot with more than 20.000.000 edits. Amir regularly tinkers with the routines that he uses. Sometimes he gets better performance, sometimes he gets a better result.

The algorithm for adding labels has changed several times and, the result of the latest change can be seen in the statistics below. You may notice several spikes, the last one is captured in the last dump; it resulted in many more labels for items where already one label existed.

It is people like Amir qho make a real difference. One bot request of his for Commons will help the Commoners see that Wikidata knows about the people mentioned in the Creator templates. Jobs like this are essential when the wikidatification of mediafiles is to succeed.
Thanks,
      GerardM

#Pywikibot - #Thanks for your #help

#Pywikibot - wants you to peek under the hood

For most #Wikimedia projects, Pywikibot has proven itself to be a trusted hard working tool. Literally millions and millions of edits were only possible because of the people operating the bot.

Before Wikidata, the interlanguage links were maintained by what was called "pywikipedia" bot. Now Wikipedia has been harvested for information with this bot to enrich Wikidata.

As time goes by, the architecture of MediaWiki has changed a lot. Consequently Pywikibot had to evolve as well. Its architecture changed as a result; it aims to use the latest API and other MediaWiki functionality.

From July 24th, 2014 and ending on Sunday, July 27th there will be a big online event to learn what more needs to be done for Pywikibot. What bugs need an urgent fix, what features are missing or incomplete. Obviously, it is a also time to look at the code and look for "bit rot".

It would be awesome when more people who care for the Wikimedia projects and know their Python get involved and ensure that Pywikibot will remain the meanest and most powerful bot platform around.
Thanks,
    GerardM

#Wikimania - a visa denied

Meeting Amir at a conference, a hackathon is a pleasure. Having him around is a sure way of getting all kinds of problems solved. Giving his intimate expertise of pywikibot and our projects he makes it seem easy to resolve issues. The biggest issue is often that it takes time for his bots to complete.

Amir has been to many conferences but we will have to miss him in London.. It took the British embassy several weeks and all kinds of excuses why it took so long to decide that they are afraid that he will not go back.

Really?

The success of Wikimania 2014 will not be as great because one of our best and brightest will not be among us.
Thanks,
       GerardM

#Wikidata - Rejoicing for new rollback functionality

When Wikidata is to be relevant as a source for info-boxes, it has to include substantially more information. For humans it starts with really basic information.. human, gender, date of birth/death, nationality, spouse, child, occupation, office held.

It may sound odd, but once you have data, you can manipulate it. Without data you are stuck. Take for instance the Olympics, originally all the athletes were amateurs and the current practice is to make all of them professionals at their game.

You can imagine that to get much LOTS of data in, things go wrong occasionally. Consider for instance the directory "Brazilian people by occupation" or "American politicians" with a name like that you expect all the humans mentioned in those categories to be Brazilians or politicians.. They are not.

I am really happy that Autolist2 allows for rollback; it took care of expatriate footballers in Brazil. I am really happy with friends like Amir he took care of the problem with all those card carrying Americans. He even promised to make it a pywikibot program once he finished his exams.

Those expatriate footballers live or lived as residents in Brazil, those Americans may be Republican or Democrat. That is another opportunity for data inclusion. However, when you enter data wholesale, chances for lots of problematic edits are a reality. Statistically maybe not that much but enough for them to be noticed and, that is a good thing.

Rollback functionality is essential to maintain the quality we all seek. It allows us to make mistakes and make amends.
Thanks,
     GerardM

No individual #WMF grant for a #Wikidata economics project

One of our best and brightest is from #Iran. He is a very prolific contributor both to Pywikipedia and to Wikidata. His bot has the highest number of edits as a bot and, he is trusted to run his bot as admin.

Amir is part of a team that hoped to bring economic data to Wikidata. Their plan was sadly denied. The reasons given are imho a bit upsetting. Amir is from Iran and we cannot pay money into Iran it is said. Really? We do not have to pay into Iran, what is asked is to make funds available for a project. It does not follow that the WMF itself has to transfer money into Iran.

Another reason is that currently Wikidata does not yet support units properly.. That is in this instance not really much of an issue either because before the data is ready for upload, a lot of preparatory work is to be done. Getting the data, proposing missing properties, analysing the data you name it. Oh and yes, preparing the software that can handle this type of data.

In my opinion the decision not to grant this proposal is a mistake.
Thanks,
     GerardM

#Wikipedia vs #Wikidata - a qualitative comparison

When a Wikipedia thinks it knows something, and Wikidata begs to differ, what to do? Wikidata has its information from a different source or someone actually looked it up or someone made a booboo. It is all possible and someone has to look into it. This is where sources become actually relevant.

The thing to do is actually pretty simple. You research it or you report it. For a bot that is looking for specific facts, reporting is the obvious way to go.

Two reports by Amir show the differences for the date of birth and the date of death between the English Wikipedia and Wikidata. By highlighting these issues, it becomes possible to raise quality in both Wikidata and Wikipedia.

At this time these reports are generated for a first time. Some false positives have been found. The location of these reports is not optimal and, the communities of Wikidata and English Wikipedia are not yet aware.

Reporting on problems however is one obvious way of raising quality. It is hopefully the first of many similar reports.
Thanks,
     GerardM

The Gelman #Library, an institution in Washington DC

The Gelman library or by its full name the "Estelle and Melvin Gelman Library", is one of the many GLAMs we have content from in Commons. For all these institutions there is a template in the "Institutions" namespace.

All these templates are an amazing resource, it must have been compiled by many people over many years. On Commons it links to material that is known to be part of the collection of that institution. Making such links is in itself an Herculean job and one that is not finished at all.

The template on all those institutions typically refer to a Wikipedia article. The Wikipedia article links to a Wikidata item and it aims to include all the information that exists on the institution template on Commons. To harvest all that information tools like the pywikipedia bot will be used.

Some people complain about Wikipedia that "everything is said and done". For those people Wikidata is a bit of fresh air; there is so much more that can be done.
Thanks,
        GerardM

''#Pywikipediabot'' ''20:00 Thursday, October 24, 2013 - Sunday, October 27 22:00'' UTC

The one tool used by most bots has not only stood the test of time, it is getting ready to complete its rejuvenation. It moved closer to the tools used in the Wikimedia Foundation. The road map has it that a triage is needed.

This interview happened largely on the pywikipedia mailinglist. Another great resources for bot runners. What you read is a compilation, more takes can be found in the list archive.. :)

Enjoy,

      GerardM

What is pywikipedia?

• A Python-based framework to manipulate Mediawiki installations. Any installation, not only those run by the Wikimedia Foundation, can be worked on. You can change everything, you can change in the Wiki as on  an editor also per API. Thus, pywikipediabot can be used to create so called bot tasks, to do the same change to a lot of pages and do it fast.
• A huge bunch of scripts which use the framework above for all tasks you can think of: One script for example for mass uploading pictures (like scans of a book), one script for cleaning up page source code (like removing <br>, multiple empty lines, reorder  interwikilinks,...), scripts for fixing common errors, and so on. change everything you can change in the Wiki as an editor also per API. Thus, pywikibot can be used to create so called bot tasks, to do the same change to a lot of pages and do it fast.

As I understand it there are two versions, "core" and "compat". What is pywikipedia core and what is compat?

Compat is old. Core is the redesign with complete new and cleaner data structures. Most new API functions (like those to modify Wikidata) are only or much better supported by core.

Why keep them both, it must be a lot of work to have to maintain them both..

There are so many working scripts which do their job now for years - thus, there is not much pressure to move them to core. Nonetheless, if you plan to do something new, use core, it's much more appealing.

Recently all the bugs have been moved to bugzilla ... What is it that you hope to achieve by this?

Handling bugs will be a lot more easier, there will be more eyes to keep on debugging process, tasks will be centralized, it will be more wikified by merging in WMF infrastructure, you can see more in this suggestion page.  

At Sourceforge you have your code repository and - unlinked - your bug and feature request system which over the years get filled with a lot of dormant, outdated bugs, which were fixed a long time ago. We hope to get a clean list of bugs.

Recently all the code has been moved to GIT ... What is it that you hope to achieve by this?

GIT allows you to work much more naturally and doesn't break everything when you merge your different ideas again. So many new features compared to SVN that I do not want to outline this here. In short: In SVN you can branch your code if you do something different. But merging does never work, therefore, noone uses this important feature. With GIT, merging works (as it stores much more information as SVN). Thus, whenever you start a new task, you branch, you have a new idea.

What is the biggest challenge running the pywikipedia bot?

On the pywikibot side, learning its limitation and working around them. On the larger, bot side of things, performance. You have to always balance thread-safe generators and better support for sections (such as retrieve the whole page but submit a section or vice versa). The API/server performance with the local computer's performance (mostly disk access for me, might be RAM limitations when running on a VPS) and the network performance. While not related to pwb, this is something you have to consider for each new bot one writes. pwb could help by providing more advanced support for threads, such as retrieve the whole page but submit a section or vice versa)

How many people are using the pywikipedia bot and how many people are developing code for pywikipedia bot

It has been widely used since 2003, and It has more than 100 authors but now they are just five active developers. After the launch of Wikidata many bots lost its work :-) e.g. one bot was active on all wikipedias, now it works only on 2-4 and the wiktionaries.

It is possible to use the pywikipedia bot in so many ways... Is it easy to learn what it can and cannot do?

Yes: Just join the irc channel and ask. For nearly all tasks, some "swiss army knife" exists, and the developers are extremely helpful. Just ask.  

When script have documentation inside, its easier. Some scripts have documentation on mediawiki.org, but nobody really knows all about it.

How long does it take before pywikipedia bot supports a new Wikidata data type ?

Usually it takes a week or two based on how much the developers are willing to do. The following datatypes are already supported: item, string, URL, coordinates. Time is nearly done. - > ~ 2 month

Tomorrow there will be the pywikibot triage... What is it and who can participate?

The bug day will focuse mainly on categorizing and prioritizing and closing non-reproducible bugs and fixing them if the bug is not a big deal. Because we just migrated there are ~700 open bugs (some of them are really old and it's fixed long time ago) so we need to clean them up. It's really necessary for us to have this "bug triage".
The good news is that anybody can participate.

#Wikidata: Operation "Fat Head" is a GO

New functionality for #Pywikipedia has been writen. When you use search in Wikidata, there were many missing links because there was no label for them.

Operation "Fat head" is a job executed by Dexbot it uses the names of Wikipedia articles for the new labels. They are used for searching Wikipedia so they will work equally well for Wikidata.

I expect that the most requested items will benefit the most and, as a result Wikidata will gain relevance in many languages.

Thanks,

      GerardM

About #political parties in #Wikidata

Adding data to Wikidata does not have much of a priority. Yes, there are some people who run bots but when you can only run bots and rely on the quality of the programming it is not nice.

Within the pywikipedia.bot environment it is the harvest_template.py that has the most potential. At this moment I am running a script that adds one political party to people who have the template "Infobox Officeholder" on their Wikipedia article. These are people like Nancy Pelosi or Moritz Leuenberger I am adding the first political party mentioned for them.

I am really happy that I am running a hacked version of the software; two bugs that are show stoppers for the autonomous running of the software have been fixed in my version of the software. I can only hope that some real work will be done. A list of all the problematic content would be nice.
Thanks,
      GerardM

Reviving my bot for #Wikidata IV

An appropriate response of a fixed bug is .. more testing. The harvest_template.py did get a bug fixed so testing the latest and greatest was in order. The bug has been fixed. I am grateful so it was time to do run attempt to import more data from Wikipedia and do some more testing at the same time.

At this time I am importing political party information from templates of obvious politicians; people who are for instance in a category like "Presidents of Argentina". I found new bugs..

• When there is no party indicated, the bot aborts
• When the party is a "red link", the bot aborts
• When there are multiple parties indicated, it only adds the first one

I also found that there are Wikipedia articles without a Wikidata entry, they are ignored. 

I am really happy with the help I get in running the pywikipedia software. It becomes however obvious that importing data from Wikipedia into Wikidata in this way is not mature.

Thanks,

       GerardM

PS I imported some 400+ records while testing :)

Reviving my bot for #Wikidata III

The objective of running a bot is to achieve an objective. It used to be adding interwiki links, now it is adding data to Wikidata.

It is a good idea to start small; a small group of records and only one value. If only to be able to fix things by hand without too much trouble. One bug found, fixed and up to the next challenge.

The presidents of Argentina probably are also likely members of a party so they were next. No record changed, so I checked Juan Perón to find that he was a member of the Justicialist party.

I found another bug. This bug is now being fixed as well; people are really helpful. Probably they want more data in Wikidata as well.
Thanks,
      GerardM

PS I have also done some work on the "kings of Nekor"

Reviving my bot for #Wikidata II

It did not work as designed. In the end there were several mysteries. To solve it, I deleted the user.config.py and it was re-created by the software.  That made a difference. In the end I also got the order of the parameter and the property right.

python pwb.py harvest_template.py  -family:wikipedia -lang:en  "-cat:Presidents of Uruguay" "-template:Infobox Officeholder" -namespace=0 party p102

What my bot did not get right is the attribution. It is said to have been imported from the Northern Sami Wikipedia ...
Thanks,
       GerardM

PS thanks again to so many helpful people :)

Reviving my bot for #Wikidata

Wikidata needs more data and, to eat that dog food I am in the process of reviving my bot, RobotGMwikt. I used it a lot in the past to create interwikilinks for Wiktionary. This is a function Wikidata could serve for Wiktionary as well as a start..

Things have changed; I have a new laptop and the Pywikipedia framework is now developed using GIT. There is documentation but it deserves an update. It seemed for instance that there are four things to install but actually there are three. I do not know why there is something about a "clone".

The good news is that at the pywikipedia IRC channel there are people quite happy to help. You have to start for instance in the folder with the "core" software and, this is where the user-config.py has to be as well. And, make sure that you have all the quotes at the right places. How did you copy and past again from a terminal session ... I am grateful for their patience.

I ended up with the following command:

python pwb.py harvest_template.py "-cat:Presidents of Uruguay" "-template:Infobox Officeholder" -namespace=0 p102 party

It did exactly nothing and I have no clue why. I saw all the president of Uruguay being processed and I know it is in the template and, p102 is the party... 

Please help because Wikidata needs more data.

Thanks,

        GerardM

#WMDEVDAYS - #OmegaWiki vs #Wikidata

It has been wonderful to see the enthusiasm in Wikidata. It is wonderful to notice how Wikidata is developing rapidly. It is interesting to look at what it does and, how it compares to the first, the original Wikidata project, OmegaWiki.

The short answer is, that there are three main differences;

• OmegaWiki uses a lexical perspective where Wikidata uses a concept like perspective
• OmegaWiki is more mature
• Wikidata has the support of the Wikimedia Foundation

Wikidata starts with the existence of Wikipedia article(s). Each one of those represents a concept. Berlin (Germany) is an example and  Berlin (Connecticut) is another. In OmegaWiki they would both be known as Berlin while in Wikidata they may both have an additional label for the English language with "Berlin" as its content.

This difference or perspective gives Wikidata an immediate application; it will replace the "interwiki links" for Wikipedia. One important benefit will be that there will be only one instance where these links are maintained. In the case of Berlin (Germany) a new article about this city will not see an update in 205 other Wikipedias. 

Both Wikidata and OmegaWiki are able to present details on Berlin. All the details at OmegaWiki have been added by hand. Most of the details Wikidata will know are likely to be harvested from info boxes or from DBpedia. What will also make a deciding difference is that the community interested in the Wikidata content will be huge.

So OmegaWiki is more mature... in practical terms it means that it started a long time ago and is now a bit dated. The Wikidata folks know OmegaWiki, they benefited from the experience and while they do not include all the data OmegaWiki has, there is nothing stopping them at a later date.

Many people at the Berlin hackathon are keenly aware of the disturbing potential of Wikidata; no more need for the pywikipedia interwiki bot. Information in info boxes change without a trace in the local project. But they also see the possibility of providing information in a language where its Wikipedia does not have an article... information with an infobox..

It is the support of the Wikimedia Foundation and more importantly its communities that will make Wikidata a success. This is a good thing because the inclusion of structured data in Wikipedia will prove a big improvement in data quality and accuracy.

Thanks,

        GerardM

#Statistics do motivate at #translatewiki.net

AnakngAraw provided us with proof that statistics motivate. She is one of our hardest workers and localises for a Tagalog public. Her complaint was that the daily updates of statistics did not happen any more.

Keeping our community happy is really important for us and even though Siebrand is really busy making a difference at the Wikimedia Chapter conference in Berlin, he found the time to look into it.

One of the affected statistics is the "group statistics", it show what percentage of defined groups are localised in a language for MediaWiki. Statistics like this are generated by a pywikipedia bot and the software was broken. With a quick hack it is fixed locally and our numbers are up to date again.
Thanks,
      GerardM

#pywikipedia localisation update

The #MediaWiki bot platform of choice, pywikipedia is being localised now for half a month and, the results are encouraging. When we started only four languages were completely localised. Now there are twenty.

Pywikipedia is software is software that runs locally on a computer. As the localisation find their way into the software, people will need to update their software for it to make a difference.

It is really helpful when bots provide an adequate reason for an edit. You can help your community by localising the pywikipedia messages at translatewiki.net. When you run a bot, you can improve your service by updating your software to make use of the latest and greatest localisations.
Thanks,
      GerardM

pywikipedia, the biggest #Wikipedia tool gets #localisation

Many people operate bots to do all kinds of funky mass edits in languages that they have no hope of understanding. It works out really well because many things are just routine and all it takes is for someone to operate a bot.

So far, the edit summaries were added by programming a text in the software and with over 270 Wikipedias this is not really practical. It has therefore for a long time been a wish to get all such texts to translatewiki.net.

The Amsterdam hackaton brought the right skills mix together. Mikael Nordin made the illustrations from Sweden and there was someone doing the pywikipedia bits, someone doing the translatewiki bits and now someone is promoting the notion that you can be the someone to localise pywikipedia wherever you are.

The texts that you localise for pywikipedia are texts that will be in your face in the recent changes. They will be in your face on whatever MediaWiki wiki that uses the power of many years of bot development.

Hackathons are wonderful; when people get together and work for a few days together they get so much done. The Amsterdam hackaton was dedicated to GLAM related stuff and good things got done.

Brilliant is the realisation that we have so much talent in the Netherlands. We do not need to bring people from abroad to have a really effective event.
Thanks,
      GerardM

RobotGMwikt

Several years I have run as a service the interwiki.py function of the pywikipedia bot software on all the Wiktionaries. I had up to six instances of the bot running concurrently on different projects. The principle behind it was simple; when a word in one project is exactly the same, a link is to be created. This function was created by Andre Engels and he helped me out a considerable number of times.

I stopped running the bot when people started to question the algorithm used by the bot. As I run it as a service, I was interested in changing it only when good arguments were provided why it should be changed. These arguments were never forthcoming, the reason why I stopped the running of the bot was because I was told that it is now ran from the tool server.

Yesterday, I learned that the tool server functionality although maybe really clever and efficient is only used to update the English language Wiktionary. Yesterday I was asked by Spacebirdy to run the bot again because there is a need for it. So I asked on IRC if there was a problem and I was told there was not.

Today I was told to change the algorithm because the English Wiktionary wants to create interwiki links to redirect pages and as it is a "community policy" I was told to abide by it. I asked for arguments why this made sense and no good arguments were forthcomming. It was even conceded that it would be best to discuss this in the village pump.

There are good reasons why a redirect should not be referred to by interwiki links:

• Wiktionary aims to include all words in all languages. A word spelled incorrectly in one language can be correct in another
• The specific word does not exist yet in the other project
  
• The notion of homonymy is ignored
• There are a substantial number of redirects that exist as a result of a conversion

Interwiki links are one of the few things that connect the different Wiktionary projects. It is essential to consider its use in the scope of the whole of Wiktionary and not within the narrow understanding of single projects. As more Wiktionaries became isolationist in their point of view it became increasingly time consuming and frustrating for me to run the bot. Given that I have in OmegaWiki the perfect solution for the need for interwiki links, I have no need for this functionality in the first place.

What I find disappointing is that the quality of service that I provided is no longer there. It is another fine problem that a community / project council could deal with.
Thanks,
GerardM