Wednesday, August 24, 2005

Me, CAT tools en Microsoft

When you are not a translator and when you have no dealings with the translation business, you would not expect that anything that I can do could threaten business interests of Microsoft. I was therefore very suprised when I was informed that my wish for a reference implementation of a translation glossary and my wish to integrate such a functionality in open source CAT tools like OmegaT do just that.

I am actively looking for money to enhance OmegaT. Not because I am likely ever to use it, but because I want quality translations in Ultimate Wiktionary. This reference implementation will do exactly that. In order to make OmegaT less user-feindly, there several little things that can be done. Sabine, defined some quirks that are a barrier for newbies. She also identified that "Trados" compatibility would be really important to introduce many translators to OmegaT.

A friend of mine is working on three annoying things. He is a professional programmer. He will inform us how much time it costs and how much it would cost if 100 translators payed for this. The point that I want to drive home is, that you can either pay for a license or you pay for functionality. When you pay for functionality, it will prove to be much cheaper.

I have my reasons why I want OmegaT to be a success, Microsoft happens to be the monopolist in the translation/localisation business. It is a genuine suprise but when you think about it, it should not be.

Thanks,
GerardM

Sunday, August 21, 2005

Working on the Logos website

You may apreciate that I have been spending time to learn how the Logos interface works. As the data has been worked on only by professional translators I am honoured that I am allowed to be an editor for the two languages I know best. With the content that I have available to me in Wikionary many translations in many languages are available to me. I have been adding words and translations like "gebarentaal" and the most recent one I worked on was "vakman". Now vakman is a tricky word; it is very imprecise in the translation and it has five different plurals. Now I have been looking how to deal with that in the Logos website but also to understand how it can be done in the Ultimate Wiktionary.

The differences between the current Wiktionary and Logos are profound. In Wiktionary anyone, even anonymous users can edit almost everything. In Logos an anonymous user can add words that are checked. When you are a professional, you edit translations in the languages that you know best. The thing that I missed were the talk pages; a place where you can discuss an individual word. I missed the IRC channel where I can discuss issues about words or meanings.

I understand the differences, they make sense because it shows where you are coming from. Logos provides very much a tool for translators by translators . Wiktionary is very much a tool for people who care about words/lexicology and share this in their mailinglist, IRC-channel and talk pages.

What kind of a community would result when these two communities were to merge? What kind of content? It would be an intersting experiment.

Thanks,
GerardM

Tuesday, August 16, 2005

Sign languages

Since Wikimania one of the things I am hopefull of is the inclusion of sign languages and oral languages in the Ultimate Wiktionary. Thinking of sign languages for me is really difficult; I do not sign I am not deaf and I do not know anyone who is. Wolfgang gave me an impression of what it is to be deaf; to me it is clear that the only thing I want to do is to create this resource that is there for them as well. It has to be their resource. It has to be their dictionary, they have to create the movies, categorise the content so that it can be found.

Consider what is going to happen when we can Free all the recordings of sign languages that exist in many universities and combine them with the content that we hope / expect to have, it will be a lexicological resource that will be awesome. Because of its scope it will get a relevance of its own and that is why it so great that an organisation like the Wikimedia Foundation will host it; it is not party in any of the rivalries between organisations or institutions it is just there to provide information to all the people of this world in their own language.

Thanks,
GerardM

Sunday, August 14, 2005

Priorities

Wikimania was an event. From my perspective it was great. We are going to have this weird and wonderfull thing that is going under the project name of "Ultimate Wiktionary" and given that we hope to merge our activities with the Logos project we may start with a cool 7 miljon and a bit of lemma's. We then still have to work out many details like the migration of the wiktionaries, the creation of a new community that is not logos or wiktionary but logos.wiktionary.

Wikimania was also important because it paved the way of including sign languages into the project. Wolfgang Georgsdorf gave a great presentation and together with Ascander we changed the data design to include sign languages as well. I learned that there are ISO-639 codes for sign languages as well :) .

When I came back I have worked hard to do many things that can be considered the fallout of the conference, I still have not finished to do all the things that I want to do. The nds thing did not go away and it does cost me my time. There are all kinds of things that I want to have done and I work on them. It is about priorities. Informing about what is going on is a priority, I did some work on the nl.wikimedia server. And now I finally have written here as well.

Sunday, July 31, 2005

Working on my presentations

Wikimania is only a few days off. There will be many exiting speakers. And I hope to be one. Not that I will not speak. I am getting nervous. I did my homework, I did write my presentation and there are so many things I would like to include but can't.

I can't because things have to develop some more. Because Ultimate Wiktionary is still some way off. Because people are still thinking old Wiktionary and I would only confuse most people even more.

Really, when we get UW life and functional it will be great. The filenames that I used have names like Spelling, Word and Meaning. The problem is that people think of them as a spelling a word or a meaning. Somtimes I think I should have named them Kwik, Kwek and Kwak. (you may also know them as Huey, Dewey, and Louie. It would make it more abstract but if it would help ??

Wednesday, July 27, 2005

Translating proverbs

When you put proverbs in a dictionary, you add all the usual things. You describe the meaning, you add an etymology and you translate the proverb. However how do you translate proverbs, do you translate them literally or do you want to give a proverb with a same meaning.

The proverb "de beste stuurlui staan aan wal" has a similar meaning as the English "backseat drivers". The Dutch version is nautical and the second one obviously not. So it is hardly a literal translation but it is a functional translation. In general terms the meaning can be described and as such it would function however, I can apreciate that there will be a certain drift when proverbs from many languages are put forward.

The problem for me is to consider how to deal with this in the Ultimate Wiktionary.. Well, at this moment we do not have it yet so it is not a problem .. :)

Thanks,
GerardM

Saturday, July 23, 2005

Wikimania

In a few days wikimania the first world conference of the Wikimedia Foundation will open in Frankfurt. It will be a great time for the people that are able to come. There will be some things that will be broadcasted over the Internet.. The agenda is absolutely smashing. I will be speaking as well and I will speak in "die grosse Saal". It will be the final presentation in a series of four;

*Wikidata will be about the technology that will be behind Ultimate Wiktionary and many other projects.
*Wikisign will be about creating a lexicological resource for the deaf.
*Logos will bring us what their experience is hosting lexicological content
*Ultimate Wiktionary will be about what we hope achieve in the next generation of Wiktionary

As I have been talking so much about what I hope to achieve, I may not bring you anything new. However, there is so much to it..

Thanks,
GerardM

Friday, July 22, 2005

Feedback

I am starting to get feedback on my ERD. As was to be expected, what is clear to me is not necessarily clear to others. The great thing are the suggestions that come with the feedback, things like call it "Script" in stead of "Characterset" and there is an ISO code for it. Another improvement was to include a field to say that a specific relation (eg idiom or proverb) are language dependent.

Some of these things are so basic that you tend to forget to include it by making it explicit. All in all it proves important, publishing and publishing again does work.

Wednesday, July 20, 2005

Sound and sign

I am progressing with the datadesign for Ultimate Wiktionary. The current challenge I am facing is to deal with both oral languages and sign languages.

The easiest for sign languages was the realisation that a movie is the "Pronunciation" of a signed word. This made me change the fieldname from "Soundfile" to "Mediafile". More complicated is the fact that there are some four written signlanguages. These I would really want in the Ultimate Wiktionary. The question is, do they have like Chinese does their own UTF-8 characters. When they do, I do not have to do anything. It would just work as designed.

I have realised that languages like Arabic and Chinese are formal written languages. There are many people who have a spoken language that is grammatically and syntactically (does this word exist?) different from the formal words. So when I record pronunciations, how do I deal with those. How do I register those lanuages? How do I indicate that these languages use Chinese / Arabic for their written language..

My working theory for the moment is that there may be transcriptions for those languages. Certainly when they have been noted by someone who has some authority, these can be used to link the essentially oral words with something that has characters. These characters are needed at this moment to make it possible to enter them in the database. Now the question is, how to relate them to the written language ... At this time it is just a matter of having the written word as a translation.. in effect this is correct.

Thanks,
GerardM

Thursday, July 14, 2005

Working on a table design

I am working on the table design for the Ultimate Wiktionary. I have posted the current version of my ERD and true to the tenets of Open Source I am working on it and will post often, people can deduce what I am thinking. The funny thing is that since the last time I created a more or less working model for an UW, I have learned so much. The resulting datadesign is significantly different. I have come to the realisation that what I create is very much the result of this proces of assimilation of a lots of loose ends in the current Wiktionaries.

One of the things that is funny is that when you design a database design you not only have to think of the data itself, you also have to think about how it is to be used. The problem for me is that the database and the development are pretty much divorced. I know databases pretty well but I do not know the restrictions of MySQL in combination with what Wikidata will bring us.

I find it really thrilling that we are at the stage where there is an imminent need for the datadesign for Ultimate Wiktionary..

Thanks,
GerardM

Friday, July 08, 2005

talking tables / files

We are arriving at the point where we have to talk file design. What tables do we need what fields will they have and how will this relate to the functionality of what we have.

The three most important tables will be "Language" "Word" "Meaning". They are top down related. The most difficult to understand will be "Meaning" because the meaning itself will be in a seperate table "Meaning-text". This is because the text of meaning is to be had in every language, and it is the abstraction of the meaning that is in "Meaning".

This "Meaning" will relate to synonymes and translations (a synonym is equivalent to a translation in the same language). This will give people an instant problem many words are not the exact translation of another, so how will we deal with this.

When a word is translated, the word picked in the translation is the one that fits best in the meaning of the original word. This meaning is therefore one that is of importance to this word as well. This meaning can be endemic to the language of the word, this makes it a natural fit or the meaning can be external to the language of the word. When the meaning is external to the language, this meaning is only relevant when translating the word.

This sound problematic. The word girl, meisje, Mädchen are good translations. In the Neopolitan language there are words that are specific to girls of a certain age. The meaning of these word is included in the meaning of the word girl. They need to be shown when you are interested in the Neopolitan language. However, when you are not interested, these meanings that are external to words of the English language, do not have to be shown.

I have been told that there are some four words that can be included in the word girl. These meanings do relate to each other and as such it makes sense to use thesaurus like structures to describe these relations. As these relations describe the meanings, these relations are relevant when you are interested in the Neopolitan language. They do help a translator choose the best fit and also alternatives when one word is used too often.

Thanks,
GerardM

Thursday, July 07, 2005

Supporting a "bot"

The Wiktionary projects are very gratefull to the programmers of the pywikipedia bot. Particularly Andre Engels has been important in supporting this bot for the Wiktionary projects. He has programmed new functionalities that helped us work together more than what we would have done without it.

As can be deduced by its name, most people use the pywikipedia bot for Wikipedia projects. Many of the innovations have been programmed with Wikipedia in mind. The latest innovation allows you to be logged in several projecs at the same time. The interwiki bot makes use of this facility and when it finds that one project needs to be updated, it will do so. This enhances the quality of the bot dramatically.

Supporting a non-programmer like myself is a pain. It is therefore important that tools like tortoise work well. It means that a common baseline can be created. This in turn facilitate the analasys of error conditions. Today we finally got it to work. We had to remove the application and start it all over again.. This time it did download the pywikipediabot software from Sourceforge..

Really, Open Source rocks when there are friendly people like Andre ..

Thanks,
GerardM

Wednesday, July 06, 2005

IATE and Free content

IATE or Inter-Agency Terminology Exchange is a project that is to create a glossary to be used for the European Union. It has live data and its content was appreciated by many translators until recently. Until recently, there was a guest profile with a guest password that was used by many. Because the IATE database is "not ready for the general public as it may not cope with the demand that might be put upon it", this access to the public is removed.

To gain access, you have to translate for the EU and you have to sign a contract that you use it only for EU use. There is however one bright spot; its copyright. The IATE copyright says clearly that you can have this data and use it as long as you attribute it to the institution that manages this information.

It is therefore a lucky coincidence that we want to make Ultimate Wiktionary relevant. It is as fortunate that we already plan on cooperating with the EU by publishing its GEMET content. When we have proven that we can host lexicological data, we can ask the EU if we can host this data. It is relevant data it is important data so much so that the EU expects that its modern systems will crash under the strain of all these people who want it.

With the Wikimedia servers, we are used to provide as good a service as we can. We do not promiss 0.9999 uptime, we do the best that we can. And, if this data can be had for the lexicological information that it is, we are quite happy to host it. We are quite happy to cooperate with the EU to make this information available and more relevant then it is at the moment, being a "secret".

Thanks,
GerardM

Sunday, July 03, 2005

The need for a reference implementation

To make Ultimate Wiktionary relevant, we need data, we need a big community, we need relevance. Relevance can be had in several ways. One way in which we may get both more people and more content is by making the content of UW as a translation glossary to be used in translation tools. There are several translation tools, Sun Microsystems Opened up its CAT tool, OmegaT is another and, there are more.

The functionality that all these tools will derive from UW is the same. So having an implementation that provide the bare bones of what is needed makes sense. It does help to make a bigger group of people aware of the wish for this cooperation. It hopefully leads to the cooperation of the different communities behind these tools, in order to improve the quality of all the tools.

To communicate about the tool, we have started an experiment with Google groups. Here you find a discussion list. Everyone can read this, but only members may write to this list.. this helps against SPAM :) . It is not a Sourceforge environment yet, this is something that people who will develop this reference implementation should decide on.

It is always exiting to see how these things develop. I hope for the best.

Thanks,
GerardM

Thursday, June 30, 2005

The capitalisation of the English language Wiktionary

With some 76.000 articles, the English language Wiktionary is the biggest Wiktionary. Until yesterday all the articlenames were capitalised. Some months ago there was a vote to change it so that articles would be as the word is spelled. This decision was not implemented, a lot of words were spilled on this issue and now many months later, out of the blue it was changed.

The English wiktionary now has a problem and, it has an opportunity. The problem is that many entries are wrong. The problem is that the interproject links to Wiktionary from Wikipedia are wrong. The opportunity is that there are many other things wrong as well and it is therefore an unsought opportunity to revisit the content to improve the content.

Many people will feel frustrated because of all the huha. Many people will feel angry because the timing was not great; it stopped the migration of Wikipedia to release 1.5 temporarily among other things. But as the opportunity is there. It is also the time to step to the plate and do the best that can be done.

I am speaking to Andre Engels and I hope that he will come up with a bot that will find the capitalised words and move them back to capitalisation. This bot should also be able to list the words where a word can be found in both upper- and lowercase. After this the interwiki.py bot can be run again.. There is also a need for a bot that checks the en.wikipedia content for links to wiktionary and checks if the article is there and if not fixes it to undercase..

Yes, there was a need to prepare this change but it is also understandable that given that the decision was reached so long ago it could go wrong as it did. So now we have to do without preparation and just do the work ..

Thanks,
GerardM

Monday, June 27, 2005

To what length do you need to go to convince people

I am propably the official spokesperson for the Ultimate Wiktionary. It is this default thing; I came up with the idea and I carried it forward, found the needed funding and I do a lot of the evangelising. When I started with UW I had to learn a lot and I did. I do not expect that I have finished learning but anyway ..

There is this thing with convincing people, to what length should you go. To what length do you want to go to make people buy into an idea? My time is valuable in that I can spend it only once and if I spend too much time arguing I do not speak with people who have idea's on how things can be done. Arguing does not help the project when the result is not positive.

There are people who insist that I do everything by IRC or e-mail while I prefer to skype as it gives me better feed-back. There are people who are only interested in a tiny specific part of Wiktionary.. only English seems to be relevant to some. There are people who think they quote me and say things I would never say.

Basically, to me there are three groups. People who understand what I am saying, people who want to understand what I am saying and people who for whatever reason do not want to hear what I say or cannot understand what I say. With the first two groups I can talk. We do not have to agree but there is this basis of understanding. With the last group who can be quiet vocal, I find that they waste my time.

The problem is, there is always the off-chance that it is me who does not hear what they are saying. It may be a dilemma where there is no good solution. When I can adress the problem why they do not hear or understand what they say they may become part of the people that become relevant... So, how much time to spend on this and how much to spend on new things.

Spending time on new things is a hazard in itself. It moves me even further away from the people who find it hard to hear / understand what I am on about..

I think I will add the word "frustratie" to the nl.wiktionary..

Thanks,
GerardM

Sunday, June 26, 2005

Great news on licenses

Today, Jimbo Wales told me that I can quote him; "the license will not prevent Open / Free software projects to use our data". This is indeed good news. It means that we can host data and cooperate with every and all. It means that we can host data for organisations that are less well equiped to do this.

When we can pull it off to do these kind of things we will add extra relevance to the Ultimate Wiktionary.

Thanks,
GerardM

Saturday, June 25, 2005

Old spelling / new spelling / alternative spelling

When you think about an Ultimate Wiktionary, the idea of including all words of all languages is a given. That is ambitious enough. You do not need anything more, right ?

The Dutch language will change in 2006, it will change things that are artificial like paardenbloem back to paardebloem, it has always been pronounced as paardebloem.. The result will be that many words will be wrong from 2006 onwards.

In October 2005 a list of words will be published with the old and new spelling. It means that we have to cater for this list in the Ultimate Wiktionary. So the Ultimate Wiktionary has to be more ambitious alas..

This is then the time to start experimenting. So I am using the word Imbiß as an example, in modern German it is spelled as Imbiss, I have introduced two new templates. One to be used in front of everything to signal old spelling and the correct one. One to say that it used to be correct.

Having a date for the change will make the information even more valuable. When UW is used within software to be used for optical character reading, it may be used as a pass after the initial pass that did the scanning. It will allow for an appropriate spellcheck that will allow to enhance the quality of the OCR process.

One thing to consider as well is that some spellings are local to a certain region or country. Rudolf Heß is called Rudolf Hess in Switzerland.. the "scharfes S" is not used in die Schweiz.. So words that still have there "scharfes S" in German, are spelled differently in Switzerland. This is just spelling. Some words or their meaning are not known to all people who speak German like "Paradeis" which Austrians know to be a "Tomate".

I am more and more appreciating the fact that linguist find it astounding that we attempt to make the Ultimate Wiktionary a reality. What makes us try it is that it was for us a natural growth path from Wiktionary. So we have our problems serially and not in a parallel fashion. The issues are there to be solved and they can be solved. Getting the issues serially helps because it prevents you from being overwhelmed by complexities for us it is just a matter of refactoring.

Thanks,
GerardM

Tuesday, June 21, 2005

The value of computer conferences

I have written earlier about the Holland Open software conference, I was happy to be there and gave a presentation there as well. A conference like this is really valuable, you make contacts and when you have a new project like Ultimate Wiktionary these are really valuable. They may alter the way a project is run. One such contact I had with Mr Bart Knubben of OSOSS. This organisation is about Open Standards and Open Source Software in the Dutch government.

OSOSS is working hard to make the list of properly spelled words maintained by the NTU available for the public. Because of all kinds of contractual restrictions this is not possible at this time. To alleviate this issue, they are working as the focal point of the Dutch Open world to get the list of the NTG, the Nederlandstalige TeX Gebruikersgroep, validated for the spelling. This means that some 222.872 words will be validated.

This list of differently spelled words, comes with indications how the word is to be broken up at the end of a line. When the UW is to host such a list, it will mean some adaptions to the software; we will want to keep track correct spelling. As the Dutch spelling will change in August 2006, it means that we will want to retain the old spelling and mark it as such. As the change of the spelling rules will be in the future, we will have to consider how to deal with this.

When we host a resource like this for the NTG, it means that our license has to be compatible with the NTG. Currently they use the GNU Lesser General Public License. They do not care who uses it under what license as long as it stays Free.

Technically there is this issue; we want to host this data for the NTG. It would be really cool to be the resource for the Open/Free content world and host the Open/Free resource for the Dutch language. It would very much be in line with our objectives. We will find a solution for this issue; one thing is sure the LGPL is not applicaple for a wiki. :)

Thanks,
GerardM

Monday, June 20, 2005

What do you need in a dictionary item

When you have an entry in an electronic dictionary like Wiktionary, what is enough to make it worthwhile to have it? The question is relevant as there are people who are of the opinion that any article that does not have extensive defenitions and etymology is substandard.

My opinion is a bit more inclusive, I would like to have extensive definitions and etymology but for me the sheer fact that a word is properly spelled is enough to have it in an electronic dictionary. The Dutch language knows an institution that does provide the authorised list of correctly spelled Dutch words. For me a list with these words would be a worthwhile contribution to the Ultimate Wiktionary. Obviously, it would be a bit meagre but it does serve its purpose.

When the correct way of spelling words changes, like it will do on the 15th of October, an electronic dictionary has a clear advantage over paper based dictionaries. It is however not clear to me how We should cover the old correct spellings. In a way it is relevant to have a history of correct spellings. It could/should be part of the database..

Thanks,
GerardM