Batch-remove empty lines at the end of many Confluence pages

In a customer project we’ve decided to collaboratively write a bigger bunch of documentation in Atlassians Confluence and export that with Scroll Office, a third-party Confluence plugin, into Word.

That worked fine so far, but soon we figured that we’ve been kind of sloppy with empty lines at the end of each page, which were obviously taken over into the final document. So instead of going over each and every page and remove the empty lines there, I thought it might be easier to directly do this on the database, in our case MySQL.

The query was quickly developed, but then I realized that MySQL had no PREG_REPLACE function built-in, so I needed to install a UDF, a user-defined function first. Luckily, this UDF worked out of the box and so the query could be finalized:

UPDATE BODYCONTENT 
JOIN CONTENT ON CONTENT.CONTENTID=BODYCONTENT.CONTENTID 
   AND CONTENTTYPE LIKE "PAGE" AND PREVVER IS NULL 
SET BODY=PREG_REPLACE("/(<p>&nbsp;<.p>)+$/", "", BODY) 
WHERE BODY LIKE "%<p>&nbsp;</p>";

This query updates all current pages (no old versions) from all spaces that end with at least one empty line <p>&nbsp;</p> – this is Confluence’s internal markup for that – and removes all of these empty lines from all matches pages.

This was tested with MySQL 5.5.35, lib_mysqludf_preg 1.2-rc2 and Confluence 5.4.2.

I don’t need to mention that it is – of course – highly recommended that you backup your database before you execute this query on your server, right?

Access the Android menu in VirtualBox on a Mac host

If you’re desperately trying to get the Menu button in an Android x86 installation working under VirtualBox on a Mac OS X host – whose keyboard of course doesn’t have this “context” / “menu” key Windows keyboards have on the right – you might find the touch-only-device mode in Android x86 handy:

  1. Click on the clock in the status bar to enable / disable this mode altogether
  2. A swipe from the left to the right emulates the Menu button function
  3. A swipe from right to left emulates the Back button function
  4. Simply clicking on the status bar brings you to the Home screen

(Source)

Einführung die Versionsverwaltung mit monotone

Mathias Weidner hat eine deutschsprachige Einführung in die Versionverwaltung mit monotone veröffentlicht. Er behandelt darin die Grundlagen der verteilten Versionsverwaltung, die ersten Schritte mit monotone, sowie die täglich anfallende Arbeitspraxis mit einem Versionskontrollsystem. In späteren Kapiteln widmet sich Mathias auch erweiterten Themen, etwa wie monotone auf die individuellen Bedürfnisse des Nutzers angepasst werden kann und liefert weitere nützliche Hinweise für Ein- und Umsteiger von anderen Versionskontrollsystemen.

Das Buch ist CC BY-SA 3.0 lizensiert und kann als Paperback über lulu bezogen werden. Eine Vorversion in PDF-Format ist ebenfalls verfügbar; da der Autor die Quellen selbst in monotone verwaltet, sollten auch diese demnächst verfügbar sein.

Vielen Dank an Mathias für seine Bemühungen!

Taxbird nun in MacPorts verfügbar

Der ein oder andere kennt Stefan Siegls kleines Programm Taxbird eventuell schon: Es erleichtert Selbstständigen unter Linux die Abgabe von Umsatzsteuervoranmeldungen gegenüber dem örtlichen Finanzamt.

Mac-User blickten jedoch bislang weitesgehend in die Röhre, da der offizielle Elster-Client bis heute nur unter Windows verfügbar ist. Da die Komponenten von Taxbird (die mit Gtk geschriebene GUI und eine Bibliothek zum Datentransfer, die auf den passenden Namen “libgeier” hört), Standard-Bibliotheken und Tools benötigt, habe ich mich an einen Port gemacht und diesen heute fertiggestellt.

Wer das ausprobieren möchte, der möge einfach sudo port install taxbird ausführen. Beim ersten Start erkundigt sich Taxbird nach html2ps, welches optional zur Übersetzung des HTML-Reports in PostScript genutzt werden könnte. Ich habe hierfür bislang auf einen weiteren Port verzichtet, da html2ps weder über eine Standardinstallationsroutine verfügt und außerdem etliche weitere Abhängigkeiten mit sich bringen würde. Die HTML-Reports sollten daher vor dem Versand einfach abgespeichert werden.

Feedback ist willkommen!

SQLite to MySQL

Migrating a database dump from SQLite to MySQL can be somewhat of a hassle. After various trials I came up with the following that worked for me quite well:

echo .dump | sqlite3 mydb.db |\
  egrep -v "\b(BEGIN TRANSACTION|COMMIT|PRAGMA|sqlite_sequence)\b" |\
  perl -pe 's/^([^'\''"]*)"([^"]+)"/\1`\2`/' |\
  perl -pe 's/\bautoincrement\b/auto_increment/' |\
  mysql -uroot mydb

The most important line is probably line 3: SQLite encapsulates all table and column names in double quotes and this is not understood by MySQL, so we’re replacing them with backticks. One must be careful here, though, because data INSERTs might also contain double quotes and to avoid having these wrongly replaced as well we only replace them until we encounter the first single quote (these are used in SQLite to encapsulate strings).

Line 3 might also have to be called several times in a row in case multi-column indexes are part of the schema as well, as it will only replace the very first, but no subsequent matches. (I couldn’t get zero-width lookbehind to work properly to remedy this problem; help is welcome.)

mtn-browse 0.72 and accompanying Perl library released

Tony Cooper announced a new release of his Monotone Browser software and also a new version of the underlying Monotone::AutomateStdio Perl library.

Both packages are now compatible with the most recent version of monotone, 0.99.1. Additionally, mtn-browse also supports all the new selector functions introduced in monotone 0.99 and is able to restrict revision and file histories to specific branches.

Many thanks to Tony for his outstanding work! I’ll update the MacPorts packages in a few…

monotone docathon on Sunday, 18:00 UTC

The monotone team holds a “docathon” – a small documentation sprint – next Sunday, Dec 19th 2010, starting at 18:00 UTC for approx. 6hrs. During this time we meet on IRC and want to collaboratively tidy the wiki pages, further improve the manual, update INSTALL and cleanup other text-based files and more for the upcoming 1.0 release.

So if you want to lend a helping hand to your favourite version control system, just drop by on IRC, say hello and we’ll quickly get in contact with you 🙂

See you on Sunday!

Why I do Open Source development

I was on a scrum master certification over the last two days, lead by Joseph Pelrine, and beside the enormous amount of knowledge on Scrum and the not lower amount of anecdotes and stories he experienced with [insert famous IT company here], he came out with one particular interesting hypothesis:

“Doing Open Source software is like having an affair.”

He argued that if people would have enough fun with their daily development tasks and would identify enough with their company, they wouldn’t have the need for doing Open Source, they would not spend their valuable time developing free stuff.

So, am I cheating my company?

Certainly I love Open Source and Free Software in general. One reason why I love it that much is because I come in contact with interesting and very smart people. For example, I learnt many things from the friendly folks in the monotone community over the last couple of years, things which I did not learn or even came in contact while doing my daily business. And guess what, my company participated and profited from this many times, because I carried a lot of this knowledge back to my day job.

Beyond learning another major thing of Open Source for me is recognition. I don’t get a lot recognition for my daily business – either because most people like my boss just don’t care how software is manufactured and how the process works (after all, its the money that counts, eh?), or because the artifact I’m developing vanishes anyway because some marketing guru of a customer decided to “well, not go into this direction any further”. Sure, the money is ok, everything is paid, but it just plainly demotivates you if you work for the trash can. With Open Source you work on stuff you or other people demand, you get the reward in terms of positive feedback if you help them solve their problems and you get public recognition for the source code you publish.

So again, am I cheating my company? I don’t think so. Open Source allows me to think outside of the box and try new things, for all of which would be no time in narrowed sprints or busy project plans, and my company profits from that. But yes, maybe this also opens the door for new employments in the future as well, but lets face it, doing Open Source work is often the only possibility to personally get ahead in this industry today.

monotone 0.99 released (Updated)

[Update] Unfortunately some last minute changes make this release break on x86_64 – the issue has been fixed in the meantime, the patch is available here. My apologies, we’ll put out a new release shortly. [/Update]

We, the monotone developers, are very proud to release version 0.99 of our distributed version control system.

This release contains almost countless bug fixes, changes and new features, so I’m listing only the outstanding ones here:

  • Selectors have been overhauled to support the calculation of common sets of revisions, like e.g. differences, ancestors, parents or children of input revisions.

  • Monotone’s URI syntax has been extended and is now the default for all kinds of netsync actions (the old syntax remains available until 2.0 though).

  • All netsync commands now come with a “dry-run” mode which shows incoming and outgoing changes, such as revisions and certs.

  • The command line UI was cleaned up and improved a lot and now supports negatable and completable options amongst other things.

  • Many automate commands have been added and existing one improved.

  • Monotone regained its manual page – automatically generated from the command tree – which is also available localized through the new ‘manpage’ command.

A complete list of changes is available in the NEWS file. The tarball can be downloaded here – binaries are posted there as they come in.

Finally, this release also marks the beginning of a new version numbering scheme, where we try to maintain stable versions up to a certain point and add only non-breaking changes as minor releases.

The upcoming version 1.0 now is such a stable release, actually the first one after more than 6 years of development. It will only contain bug fixes and documentation improvements, but no additional new features, and is planned in Q4 2010.

Thank you all for your ongoing support!

Thomas
– on behalf of the monotone team.

ACLs on a jailed ZFS volume with FreeBSD

While setting up a couple of things on the new monotone server, which is bascially a FreeBSD jail on a physical server kindly provided by Lapo Luchini, I stumbled upon a quite basic use case: I needed to give one user group read and write access to files created and modified by another group.

On a Linux system I’d have simply enabled POSIX ACLs for the volume and call

$ setfacl -m d:user:foo:rwx /path/to/directory
$ setfacl -m d:user:bar:rwx /path/to/directory

and voila, all new files and directories underknees the path would have received the default ACLs with read and write rights for the foo and bar user.

It turns out that on FreeBSD, which gained zfs ACL support in its 8.1 release recently, the call is quite similar – just that NFSv4 ACLs are used instead of POSIX ACLs, which are a bit more powerful (read more on the topic here):

$ setfacl -m user:foo:rwxp:fd:allow /path/to/directory
$ setfacl -m user:bar:rwxp:fd:allow /path/to/directory

Instead of d: for default which is used in the POSIX version, fd is used, which stands for file_inherit and directory_inherit and is exactly what we need. And while POSIX ACLs are add-only, you can either specify :allow or :deny to also explicitely deny access to a user or usergroup. Finally, you might have seen the little p which stands for the append_data right. This was needed for me in one case, so I added it here. It is actually one of many more rights beside the well-known rwx that NFSv4 ACLs defines here – if you’re curious, just read more on the aforementioned link.

“So if everything works, what is the point of this blog?” you might ask. Well, unfortunately it did not all work out so nicely. One thing caused me some headaches which is actually dubbed a feature of FreeBSDs NFSv4 ACL implementation – namely that the umask (the create mask for new files) is taken into account on file creation, which leads to an unwanted recalculation of the default (inherited) ACLs. So if you have a file foo created by user u1 and the ACLs should also give write access to user u2, then you might end up with something like this:

$ getfacl foo
   # file: foo
   # owner: u1
   # group: users
              user:u1:--x-----------:------:deny
              user:u1:rwxp----------:------:allow
              user:u2:rwxp----------:------:deny
              user:u2:rwxp----------:------:allow
               owner@:--x-----------:------:deny
               owner@:rw-p---A-W-Co-:------:allow
               group@:rwxp----------:------:deny
               group@:--------------:------:allow
            everyone@:rwxp---A-W-Co-:------:deny
            everyone@:------a-R-c--s:------:allow

The file was created with rw- --- --- (i.e. umask 077) which lead to a :deny rule for user u2 and since deny rules take precendence over allow rules, u2 gets no access to foo. I contacted the author of FreeBSD’s NFSv4 ACL implementation, Edward Tomasz Napierała, and thankfully he drawed a way out of the mess:

Your problem is umask. When you create a file, ACL entries are inherited, and then the new ACL is modified according to some (pretty complicated) rules. You probably want to disable that, so that entries are simply inherited, without further messing and adding deny entries. To do this, set aclmode=passthrough and aclinherit=passthrough ZFS properties.

The only downside to this approach is that you cannot change these settings inside the jail, but only from the outside. Luckily, the host part is also under our control, so Lapo remounted the jail volume with the new settings, and voilá, the ACLs now work as expected.

You can learn something new every day.