How to use trads.rb to manage translations and generate a Lead Translator's page

1.  Introduction

Every lead translator has his own way of managing the documents and the translations. Most of us have written a script in perl, python, or whatever to generate a page which lists documents and some information like revisions, status... Mine is called trads.rb and as the name suggests, it is written in ruby. It now helps with the translations management using command-line and web interfaces, and generates a publicly available page that all team members can use to track updates. See how the generated page looks.

The list of documents is kept in an xml file along with some related information like who is in charge of the translation or the revision number of the translated documents. A command-line tool lets you list documents, their revision numbers and whether their translation needs an update, and update their information. A web interface lets you view the list of documents in your browser. It allows you to mark them as "in sync" meaning that the translation is up-to-date. It can also display nice coloured diffs between the translated revision and the current one similar to the diffs displayed by ViewCVS. The main differences with ViewCVS is that it will display the changelog and the diff on the same page, it allows you to mark the document as "in sync" and you don't need to wait for the web nodes to sync their data.

2.  What is displayed on the generated page?

The text above the table explains what is displayed in it.

The text below the table explains in a few words how to contribute and particularly how to translate.

The table itself lists all documents defined in the parameter file (more about that later) with the following items:

• Revision number and date of last modification of original doc with links to online version and to xml source in our cvsweb.
• Revision number and date of last modification of translation with links to online version and to xml source in our cvsweb.
• Revision of original version on which translation was based.
• Link to the cvsweb diff between current original version and version which had been translated. This helps finding out what has been changed since the latest translation.
• If used, links to local html page (read on dev.gentoo.org where this page is located) and xml source in cvsweb of the original that is being translated. This helps a translator if the original is updated while he is busy with the translation and he does not want to be bothered with the latest changes yet. If you don't like this, don't use it ;-)
• The nickname of the translator currently in charge of the document.
• Priority using coloured dots.
  • Green = translation up to date
  • Blue = orginal version in col-2 has been assigned to translator in col-1
  • Yellow = document is considered nice-to-read
  • Orange = important document
  • Red = critical document
  If you do not agree with me on this yellow-orange-red classification, feel free to change it. Furthermore, the number of yellow-orange-red dots depends on the number of revisions between the one that was translated and the current one. The more revisions, the more dots are displayed with a maximum of four. This indicates how old the translation is.

3.  Getting the files

Online

You will need the following directories and files in your own ~/public_html on dev.gentoo.org. You can copy them from my own directory.

/home/neysx/public_html/images/sponsors/sevenl.gif
/home/neysx/public_html/images/gtop-www.jpg
/home/neysx/public_html/images/tek-gentoo.gif
/home/neysx/public_html/images/store.gif
/home/neysx/public_html/images/phpa-gentoo.gif
/home/neysx/public_html/images/blue.png
/home/neysx/public_html/images/green.png
/home/neysx/public_html/images/red.png
/home/neysx/public_html/images/yellow.png
/home/neysx/public_html/images/orange.png
/home/neysx/public_html/images/white.png
/home/neysx/public_html/images/empty.png
/home/neysx/public_html/css/main.css

On your machine

Please have a quick look at gorg's tiny doc page. All you need to do is run emerge gorg, then create its /etc/gorg/gorg.conf. Simply copy /etc/gorg/gorg.conf.sample and edit the root parameter.

Then, download the following archive to the directory of your choice and expand it with tar vzxf trads-200604071846.tgz

Note that it does not contain any top directory
Untar in a directory of your choice
lrwxrwxrwx neysx/users       0 2004-04-14 23:24:01 trads -> rb/trads.rb
drwxr-xr-x neysx/users       0 2006-04-07 18:28:59 rb/
-rw-r--r-- neysx/users    7060 2004-04-25 23:38:42 rb/t_log.rb
-rw-r--r-- neysx/users    1452 2004-04-07 12:38:29 rb/t_rev.rb
-rw-r--r-- neysx/users     877 2004-04-25 23:14:26 rb/t_run.rb
-rw-r--r-- neysx/users    8942 2004-05-19 00:57:00 rb/t_web.rb
-rw-r--r-- neysx/users    9725 2005-02-06 12:33:32 rb/t_opts.rb
-rw-r--r-- neysx/users    8632 2006-02-13 13:34:41 rb/trads.template.rb
-rwxr--r-- neysx/users   44754 2006-03-16 14:42:27 rb/trads.rb
drwxr-xr-x neysx/users       0 2005-04-09 01:10:22 css/
-rw-r--r-- neysx/users    3432 2005-04-09 01:10:22 css/t_web.css
drwxr-xr-x neysx/users       0 2005-11-26 15:09:45 dtd/
-rw-r--r-- neysx/users    1933 2005-11-26 15:09:45 dtd/trads.dtd
drwxr-xr-x neysx/users       0 2004-04-26 14:16:28 html/
-rw-r--r-- neysx/users    1730 2004-04-16 13:05:04 html/t_menu.html
drwxr-xr-x neysx/users       0 2004-04-16 18:05:28 images/
-rw-r--r-- neysx/users    1406 2004-04-08 22:32:25 images/favicon.ico
-rw-r--r-- neysx/users    1134 2004-04-14 16:57:41 images/valid-css.png
-rw-r--r-- neysx/users    9042 2004-04-14 17:27:38 images/gentoo.png
-rw-r--r-- neysx/users    1917 2001-09-14 06:09:08 images/valid-xhtml11.png
drwxr-xr-x neysx/users       0 2006-03-31 23:55:30 tmp/
-rwxr--r-- neysx/users   44688 2005-06-02 12:12:20 tmp/trads.rb.old
drwxr-xr-x neysx/users       0 2004-04-27 18:52:31 icons/
-rw-r--r-- neysx/users   12780 2004-04-27 18:40:57 icons/trads-www.png
-rw-r--r-- neysx/users   12075 2004-04-27 18:29:46 icons/trads-on.png
-rw-r--r-- neysx/users   12341 2004-04-27 18:32:59 icons/trads-off.png
drwxr-xr-x neysx/users       0 2006-04-04 14:10:19 log/
drwxr-xr-x neysx/users       0 2006-04-03 12:49:45 work/
drwxr-xr-x neysx/users       0 2006-04-04 13:35:12 cache/
drwxr-xr-x neysx/users       0 2006-04-07 18:27:14 xml/
-rw-r--r-- neysx/users   25723 2005-02-06 12:34:42 xml/trads.xml.old
-rw-r--r-- neysx/users   41076 2005-10-14 16:50:21 xml/trads.xml.de
-rw-r--r-- neysx/users   23131 2005-07-26 22:18:07 xml/trads.xml.sample

Obviously, you also need ruby. It should have been installed as a dependency of gorg.

4.  Configuring it

You need to translate the column headers and text in rb/trads.template.rb. Write whatever you like in $Part1 and $Part2, and format the date as you like. These two enclose the table that the script generates. You can check the GuideXML you have written by running ruby trads.template.rb. It will simply run your text through xmllint and tell you about any malformed xml.

Then copy xml/trads.xml.sample to xml/trads.xml and define your parameters at the beginning, then list all your documents. The sample file contains the current documents. Define the sync attributes to define which revisions of the original documents have been translated. If you have that information stored anywhere, you may use the command-line interface to update them. The command-line interface is covered below.

(These should be common to all teams)
<config cvsroot  = "gentoo"
        cvspath  = "xml/htdocs"
        wwwroot  = "http://www.gentoo.org"
        langfrom = "en"
(Destination to use when uploading files with scp)
        wwwdev   = "dev.gentoo.org:public_html"
(Generate html, "no" means only xml files will be uploaded)
        dohtml   = "no"
(Name of the directory where doc/, dtd/, xsl/ are located on your machine)
        localcvs = "/home/neysx/gentoo.org/gentoo/xml/htdocs"
(Which language do you translate to ?)
        langto   = "fr"
(Name of file to be generated. Script will add xml and html extensions) 
(This file will be uploaded to your ~/public_html on dev.gentoo.org) 
(You would probably want to replace the XX with your language code)
        genfile  = "trads-XX" />

 <doc ID="x86-install"> (Name displayed in first column)
  <files  name    = "gentoo-x86-install.xml" (Actual file name)
          path    = "doc" /> (path below localcvs specified in <config>)
  <info   nick    = "neysx" (Translator's nick, emails are listed at end of file)
          prior   = "C" /> (C, I, or N. See comments in DTD)
  <rev    sync    = "1.189" (Revision of original document that was translated)
          target  = "" /> (Revision of original document being translated, value is optional)
 </doc>
(You can group documents in the generated table and also specify a subdir when
the document is not directly under the directory named after the language code:)
 <doc ID='xml tips&apos;n&apos;tricks'>
  <files name='doc-tipsntricks.xml' subdir='gdp/doc' path='proj'/>
  <info nick='cam' prior='I' lock='cam' group='GDP'/>
  <rev sync='1.6' target=''/>
 </doc>

(The label attribute is used as a title in the generated table.)
<handbook path="doc" prior="C" subdir="handbook/" label="Nice Book" >

(Define filename, part and chapter numbers)
 <hbdoc ID="Handbook x86" filename="handbook-x86.xml">
(same as for individual docs)
  <info   nick    = "neysx" />
  <rev    sync    = "1.9"
          target  = "" />

 <hbdoc ID="1.b Medium Alpha" filename="hb-install-alpha-medium.xml">
  <info   nick    = "neysx" />
  <rev    sync    = "1.5"
          target  = "" />

Then list your translators at the end of the file.

<translators>
  <translator nick='toto' email='john@sillydomain.com' />
  ...
<translators>

5.  Using the application

The old way

All you have to do is update the sync and target attributes, and run the script. It will check current revisions and dates of last modification, write the xml file, download the target xml files if you use them, transform them into html files and upload them with scp to the destination specified in your trads.xml file.

$ ./trads
(typical output)
Uploaded trads.xml
         trads-fr.xml
         trads-fr-0.xml
         doc_articles_dynamic-iptables-firewalls.xml
         proj_eselect_dev-guide.xml
         proj_php_php-upgrading.xml

The command-line interface

Run ./trads -h to see the list of options and actions. The [revision] parameters should be supplied as "M.m" e.g. "1.23". [nick] is expected to be known in your list of translators but if you write only a partial nick and it can be uniquely matched against your list, the matched nick will be used. [filespec] is matched against the filenames mentioned in xml/trads.xml as a regular expression. The [filespec] should be a partial match of the filename. Besides, you can use '%' instead of the '*' to avoid file globbing by the shell.

(Filenames containing "doc/disk")
$ ./trads list doc/disk
 
S Rev.  Sync. Who        Targ. Group       File
  1.12  1.12  ribosome         System      doc/diskless-howto.xml

(Filenames containing "disk")
$ ./trads list %disk

S Rev.  Sync. Who        Targ. Group       File
  1.12  1.12  ribosome         System      doc/diskless-howto.xml
  1.15  1.15                               doc/handbook/hb-install-alpha-disk.xml
  1.16  1.16                               doc/handbook/hb-install-amd64-disk.xml
  1.14  1.14                               doc/handbook/hb-install-hppa-disk.xml
  1.10  1.10                               doc/handbook/hb-install-mips-disk.xml
  1.26  1.26                               doc/handbook/hb-install-ppc-disk.xml
  1.16  1.16                               doc/handbook/hb-install-sparc-disk.xml
  1.18  1.18                               doc/handbook/hb-install-x86-disk.xml

(Handbook files containing "disk")
(./trads list 'ha*disk' would yield an identical result)
$ ./trads list ha%disk
S Rev.  Sync. Who        Targ. Group       File
  1.15  1.15                               doc/handbook/hb-install-alpha-disk.xml
  1.16  1.16                               doc/handbook/hb-install-amd64-disk.xml
  1.14  1.14                               doc/handbook/hb-install-hppa-disk.xml
  1.10  1.10                               doc/handbook/hb-install-mips-disk.xml
  1.26  1.26                               doc/handbook/hb-install-ppc-disk.xml
  1.16  1.16                               doc/handbook/hb-install-sparc-disk.xml
  1.18  1.18                               doc/handbook/hb-install-x86-disk.xml

Let's start with the available actions:

• List: list files with current and "sync'ed" revisions. A '?' indicates there is no translation available at the moment, '<' indicates the translation needs to be updated. The nick of the translator in charge and the revision of the original being translated are also displayed.
• Sync: mark the specified files as "in sync", i.e. their translation is up-to-date. Use '0' as a revision number to remove.
• Diff: display the log history and the diff between the revision of the original file that has been translated and the current one, i.e. it shows what has changed in the English version since the translation was up-to-date.
• Refresh: force a refresh of the revision numbers and modification dates from CVS. Useless in stand-alone mode but useful when connecting to the daemon because it will not access the CVS server each and every time it needs this info.
• Upload: generate html files of targets (revisions being translated) and the publicly available page (see above), and upload them.
• Give/take: assign a file to a translator. Take is simply a synonym for «give myself this_document».
• Free: remove any assignment from a document.
• Group: assign or remove a group name to a document.
• Lock/release: this is for experimental use. It is meant to allow a lead translator and his follow-up to lock a file one is busy with when they are both connecting to the same instance of the daemon (more about that later). At the moment it works more like a semaphore because it will not block any action. Feel free to send me some feedback on how you use or would like to use this feature.

Using the web interface

The web interface will display the list of documents and let you mark documents as "in sync" once you have updated them in with favourite editor. A quick note about the fields and buttons:

• Filter: select documents based on their filename to shorten your list.
• Hide Sync: do no display documents that are up-to-date.
• Force CVS: refresh timestamps and revision numbers from the CVS server. Use this when you know or think new revisions have been committed. The application will not access the CVS server repeatedly every time it displays the list. Obviously, we do not want to overload our CVS.
• Refresh List: self-explanatory.
• Upload Files: well, upload file to your public directory. Same as the upload action of the command-line interface.
• Stop Server: ends the web interface by killing the web server

Furthermore, I found it very annoying to have to wait for the web nodes to sync before I could get a nice coloured diff in my browser and thought it would be nice to see the log history on the same page. Well, that's what the web interface does. It will also let you "sync" a document from the diff page.

See how your browser renders the list page and a diff sample. If you don't like the way it looks or if your browser does not support the css2 stylesheet, you can always change css/t_web.css. The html is very simple and straightforward and relies entirely on a stylesheet for its presentation.

Simply run trads -w to start the web interface and point your browser at localhost:8088.

Stop the web interface by clicking on the Stop server button in your browser. You can also hit Ctrl-C if you did not start it in the background or kill it if you did.

The Daemon mode

You might notice that you can't use the command-line interface while the web one is running. The reason is simply that the xml/trads.xml file is locked. This is to avoid having to read it, parse it and create a new object every time you access it. The xml file is written whenever you update anything. So, if you could update it with the command-line interface while the web server is running, your next update in the browser would overwrite your changes done on the command-line. The solution is simple: use the daemon and connect to it both when using the web interface and when using the command-line.

All the daemon mode does is actually publish the instance of its object using distributed ruby (aka drb) which is a kind of ultra light Corba on diet ;-) You start the daemon with the -d option. It will not actually daemonize itself as you might expect, just run it in the background if you want to get your prompt back. If you run it within an xterm or similar, it will die with your xterm. Use nohup or screen to avoid that. Personally, I use a pair of launchers in my gnome-panel to start and kill it. The daemon will lock the xml file, so the only way to use the application is by connecting to it. You also use another port if you like (-d 1234), the default is 6764.

To connect to your running daemon, use the -c option.

(With the command-line interface)
(Use -- to prevent the parser from using 'list' as the port number)
$ ./trads -c -- list doc/disk

S Rev.  Sync. Who        Targ. Group       File
  1.12  1.12  ribosome         System      doc/diskless-howto.xml

(Connect your web interface to the daemon)
$ ./trads -c -w

Sharing your daemon

Another aspect of the daemon mode is that it can be shared between the Lead Translator and his Follow-Up.

The daemon will listen only on localhost, the default port is 6764. If you want to grant access to your follow-up, you will need a secure communication channel to localhost:6764. One way is to use ssh and create an account for your follow-up on your machine. An account without any login shell will be enough. Copy his public key into ~hisaccount/.ssh/authorized_keys and edit it to restrict what he can do on your machine.

(Add the following just before your follow-up's public key in authorized_keys.
The command option will prevent him from running any other
command in case you forget to define a null shell for him.
The permitopen option prevents him from port-forwarding to any other port.)
command="echo Keep Out",permitopen="localhost:6764"  ssh-dss AAAAB3N...

Then, have him run something like ssh -L 6764:localhost:6764 -R 6765:localhost:6765 -N your_box.com. As long as the daemon runs on your machine, your follow-up can use the ./trads -c -r 6765 actions or ./trads -c -w -r 6765 commands and connect to the instance that is running on your box. The ssh -R option and the trads -r option are required to allow the daemon to connect back to the client.

As long as you upgrade your box with the latest security fixes, the communication can be considered reliable. You can trust it but can you trust your follow-up? That is up to you to decide. Being both trusted Gentoo developers, I suppose it should not be considered any extra threat to Gentoo. Granting access to your daemon to anyone else (like your translators) is strictly forbidden because it bypasses our policies. If you want to grant more than is currently available to your translators, upload any file or page you feel they might need to your Gentoo home page on dev.gentoo.org or to your community site.

Log files & co.

If you are curious about what the application is doing, you can have a look at the log files in log/. They will be automatically rotated, so you needn't bother about them filling up your disk.

The work/ sub-directory is used to generate the translators page and the html files. It is OK to delete its files if you want to.

The cache/ sub-directory is used to cache the diffs and the downloaded xml files because they can take a few seconds to download and this avoids overloading the CVS server. Outdated files will be replaced automatically when necessary. Leaving those files alone can save you some time and bandwidth.

6.  Future developments

That will depend on my daily use and your feedback. Some ideas already popping up in my mind:

• Allow to add new documents and translators.
• Add features to the web interface.
• Upload the diffs and replace the link to cvsWeb on the generated page.

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.