Ich schäme mich – für das Land, in dem ich wohne und in dem ich ein Bürger bin, seit gestern noch mehr als zuvor.

(Quelle, HAZ-Artikel)

Dies ist für mich ein weiterer Grund, warum ich meine Zukunft auf Dauer nicht mehr in diesem Land sehe.

On a Linux system I’d have simply enabled POSIX ACLs for the volume and call

$ setfacl -m d:user:foo:rwx /path/to/directory
$ setfacl -m d:user:bar:rwx /path/to/directory

and voila, all new files and directories underknees the path would have received the default ACLs with read and write rights for the foo and bar user.

It turns out that on FreeBSD, which gained zfs ACL support in its 8.1 release recently, the call is quite similar – just that NFSv4 ACLs are used instead of POSIX ACLs, which are a bit more powerful (read more on the topic here):

$ setfacl -m user:foo:rwxp:fd:allow /path/to/directory
$ setfacl -m user:bar:rwxp:fd:allow /path/to/directory

Instead of d: for default which is used in the POSIX version, fd is used, which stands for file_inherit and directory_inherit and is exactly what we need. And while POSIX ACLs are add-only, you can either specify :allow or :deny to also explicitely deny access to a user or usergroup. Finally, you might have seen the little p which stands for the append_data right. This was needed for me in one case, so I added it here. It is actually one of many more rights beside the well-known rwx that NFSv4 ACLs defines here – if you’re curious, just read more on the aforementioned link.

“So if everything works, what is the point of this blog?” you might ask. Well, unfortunately it did not all work out so nicely. One thing caused me some headaches which is actually dubbed a feature of FreeBSDs NFSv4 ACL implementation – namely that the umask (the create mask for new files) is taken into account on file creation, which leads to an unwanted recalculation of the default (inherited) ACLs. So if you have a file foo created by user u1 and the ACLs should also give write access to user u2, then you might end up with something like this:

$ getfacl foo
   # file: foo
   # owner: u1
   # group: users
              user:u1:--x-----------:------:deny
              user:u1:rwxp----------:------:allow
              user:u2:rwxp----------:------:deny
              user:u2:rwxp----------:------:allow
               owner@:--x-----------:------:deny
               owner@:rw-p---A-W-Co-:------:allow
               group@:rwxp----------:------:deny
               group@:--------------:------:allow
            everyone@:rwxp---A-W-Co-:------:deny
            everyone@:------a-R-c--s:------:allow

The file was created with rw- --- --- (i.e. umask 077) which lead to a :deny rule for user u2 and since deny rules take precendence over allow rules, u2 gets no access to foo. I contacted the author of FreeBSD’s NFSv4 ACL implementation, Edward Tomasz Napierała, and thankfully he drawed a way out of the mess:

Your problem is umask. When you create a file, ACL entries are inherited, and then the new ACL is modified according to some (pretty complicated) rules. You probably want to disable that, so that entries are simply inherited, without further messing and adding deny entries. To do this, set aclmode=passthrough and aclinherit=passthrough ZFS properties.

The only downside to this approach is that you cannot change these settings inside the jail, but only from the outside. Luckily, the host part is also under our control, so Lapo remounted the jail volume with the new settings, and voilá, the ACLs now work as expected.

You can learn something new every day.

Most of the things I talked about in the earlier blog post have now been implemented, i.e. you are now able to use remote_stdio to access remote databases or databases behind an usher instance. Furthermore, a forge-level control panel for usher and monotone public key support have been added.

The documentation has also been improved, there is a separate README file which lets you get started and the documentation on the new IDF configuration variables is also exhaustive.

What is kind of missing right now are support scripts for the forge-level setup of monotone. I plan to add these and even more documentation when I set up monotone’s new home server in the next couple of weeks. Yes, you’re reading right, monotone gets a different address, a fancier bug tracker and also a nicer revision browser (thanks to indefero) and if everything goes right, we’ll even offer monotone hosting support for selected projects.

Stay tuned for more updates!

Lets start with a simple example: monotone creates a special type of certs if a user explicitely calls the suspend command to mark a particular branch – actually its head revision(s) – as end-of-life, or uninteresting. This is useful in case a certain feature branch has been merged back into the main branch and the feature branch name should no longer confuse other people. Since monotone cannot physically delete these no longer needed data due to its distributed nature, the internal revision machinery now simply ignores branches and revisions which have these kind of certs and basically hides them from the user, so they aren’t picked as update candidates when update is called and they also no longer show up in the list of branches (list branches).

Of course people might still want to see at some point of time what suspended branches are actually available, sometimes maybe to revive a dead development line or for other historical purposes. For this use case monotone has a global option --ignore-suspend-certs, which simply deactivates the automatic hiding and all commands behave as if no suspend certs exist at all. So long so good, but there was a nitty gritty use case problem with this (and other similar) functionality:

If a user permanently decided to ignore suspend certs, for example by adding a specific section in his global get_default_command_options, he could not easily deactivate or overwrite this setting again via command line. The same applied to long-running processes, like automate stdio: once a client triggered the list branches command there with the --ignore-suspend-certs option, the client was unable to switch this flag off in case he wanted to query only active branches with the next invocation.

Up until 0.48 only ugly workarounds existed, such as restarting the stdio process, deactivating the loading of hooks or temporarily commenting out the particular section in the lua file, but with the recent merge of the options branch, it became possible. Every revertable, boolean flag gained a corresponding cancel flag, which is most of the time just named --no-<orginal-flag-name-here>:

$ mtn ls branches --ignore-suspend-certs --no-ignore-suspend-certs

(Can you guess what this command will now actually do? Little hint: The last occurrence counts…)

But as I mentioned above, this is only half of the truth: Not all cancel flags are just prefixed with a “no-”, in some cases they also look completly different and in some cases they even made us rename the original option, to keep the flow of the UI syntax and to prevent the invention of ugly pseudo options (or do you think --no-norc is a good name for the cancel option of --norc?).

Before you scream “Oh my god, will I ever get used to this new options?! You broke monotone for me!” let me give you a few words of relief:

1. For common options which have been changed or removed, there is a new deprecation functionality which points you at the new option syntax.
2. monotone’s inline command help will not only show you the full syntax of the original option, but also of the cancel option name.
3. Just as partial command names are completed, option names are now completed as well. If a certain prefix has multiple expansions, all possible options are listed with a short description. (For my little example above, the smallest unique prefix for the long `–ignore-suspend-certs` option is `–ignore-`, as there is also an option named `–ignored` available, for a completely different use case though.)
4. Finally, if you’re still puzzled by all the new and changed options and general calling syntax and you want a single page which you can just skim / search over for the thing you’re looking for, I’m pleased to tell you that the next monotone version will again have a manual page, but this time its auto-generated from the internal command tree and options. Maybe I’ll tell you a little bit more about this in one of the next blog posts, if not, just try it out with `mtn manpage` as soon as 0.99 hits the streets.

I hope we’ll get a lot of feedback on this for the 0.99 release, as I firmly believe that this overhaul will make the functional and consistent foundations of monotone even stronger.

I also hope that we do not scare too many people away; we did this with purpose for 0.99 and not 1.0 or 1.1, so trust me, we don’t plan to mess around with the UI to this extent again anytime soon

Thanks for reading so far. Your comments are welcome on irc and via mail.

Imagine you have hundreds of source code files where somebody once had the great idea to add a version property into each class:

With Perl the line in question is easy to remove:

But, there is one problem: Perl processes each line of the file separately when it slurps in the file, which results in unwanted empty lines:

Then I stumbled upon this article and the solution is to set a special input separator to let Perl slurp in the file as a whole:

and voila, we get what we want:

Digging a little deeper what -0777 actually means leads us to perlrun(1):

The special value 00 will cause Perl to slurp files in paragraph mode. The value 0777 will cause Perl to slurp files whole because there is no legal byte with that value.

Another day saved – thanks to Perl!

And while we’re at it, have a look at Rakudo Star, the best Perl 6 compiler which was released just recently. Perl 6 is in my humble opinion one of the well-designed languages I’ve came across so far, so if you find some time, go over and read the last christmas special, its really worth it!

Anyways, lets begin this little series with the selector rewrite Tim merged a couple of weeks ago. Selectors are one of the main concepts in monotone to pick revisions other than by their 40 byte long hash id and are therefor very useful to “navigate” between different development lines.

Monotone up until 0.48 knows already many selectors – you can select revisions by tag, by branch, by author, by custom cert values and so on. Selectors can be combined to calculate the intersection between two single sets, like “show me all revisions from author ‘Jon’ on branch ‘my.project’” which would essentially look like this:

$ mtn automate select "a:jon/b:my.project"

A particular good example is “how can I easily view the changes of a development branch since the last merge point?”. Up until now you either had to figure out manually the revision of the merge point by looking at the output of log or use some scary construct like the following:

$ mtn au common_ancestors $(mtn au select h:main.branch) \
    $(mtn au select h:) | mtn au erase_ancestors -@-

Enter selector functions

Luckily, you don’t have to write these things anymore starting from 0.99 onwards. Give the new selector functions a warm applause!

$ mtn au select "lca(h:main.branch;h:feature.branch)"

$ mtn au select "lca(h:main.branch;h:)"

$ mtn diff -r "lca(h:main.branch;h:)"
$ mtn log --to children(lca(h:main.branch;h:))"

Test drive

There are many more of these selector functions and explaining them all in detail is out of scope here, please have a look at “composite selectors” in the nightly built manual. And if you want to have an early look at this and play around without having to compile it yourself – at least if you’re on openSUSE or Fedora – just download the binaries from our nightly builds.

This is my test setup: Linux 2.6.34, MySQL community server 5.1.46, Intel Pentium D CPU with 3.2GHz, 2GB RAM

Test MyISAM tables

The table definitions are copied and adapted from the aforementioned article:

Test Archive tables

Since MySQL’s Archive engine does only support one index which is primarily used for identifying the primary id, I left out the indexed versions for that:

Test data

I re-used the procedure to create about 8 million test data records spread randomly over the complete partitioned area and subsequently copied the generated data to the other tables:

Test query and the results

I used the same query to retrieve data from all of the tables:

and these were the results (mean values of several executions):

These results were actually pretty suprising to me, for various reasons:

• I would not have thought that intelligent partitioning would beat an index on the particular column by saving the hard disk space for the index at the same time (roughly 1/3 of the total data size in this test case).
• The values for `myi_no_part` were actually better than expected – I would have thought that these should be much worse, also if you compare them with the values from the author of the original article.
• The archive engine adds actually nothing to the mix, but disadvantages. Maybe my test case is flawed because I “only” tested with 8 million rows, but one can clearly see that a partitionated MyISAM table beats a partitionated Archive table by more than 40%, so the usage of the Archive engine gives you no advantages, but only disadvantages, like being not able to delete records or add additional indexes.
• Apparently partitioning and indexing the column in question is slightly slower instead of faster, however if one tries to use a subset of a partitioned table (like restricting to where c3 > date '1995-06-01' and c3 < date '1995-08-31') it is faster - ~0.3s with index vs ~0.7s without index.

Conclusion

MySQL's partitioning is a great new feature in 5.1 and should be used complementary to subtle and wise indexing.

Now monotone (from 0.99 onwards) gained a new commit hook which is called before the commit message processing takes place, but after the logic validated the changeset and branch to which it should be committed. Its named simply validate_changes and takes two parameters, the revision to be committed as full text (parsable in the hook via parse_basic_io) as the first and the name of the branch to which the revision should be committed as the second. Just as validate_commit_message, it is expected to return a tupel containing a boolean which denotes if the change is valid or not and an optional string which explains the reason if not and which is displayed to the committer afterwards.

With this new installment, it should feel natural e.g. to create a pre-commit hook which ensures that none of the patched or added sources contains Windows line endings:

function validate_changes(revdata, branchname)
  local parsed = parse_basic_io(revdata)
  for _,stanza in ipairs(parsed) do
    if stanza.name == "add_file" or
       stanza.name == "patch" then
      local file = stanza.values[1]
      if not guess_binary_file_contents(file) then
        local fp = assert(io.open(file, "r"))
        local contents = fp:read("*all")
        fp:close()
        if string.find(contents, "\r\n") then
          return false, "CRLF detected"
        end
      end
    end
  end
  return true, ""
end

Unfortunately its not yet possible to call mtn_automate, the lua interface to monotone’s automation commands, from hooks like this. Then we could have saved the read("*all") call and would only have to scan the output of automate content_diff, which should be a little faster than doing a full string search in lua for bigger files. We, i.e. the monotone devs, are aware of the problem though and will come up with a solution sooner or later.

I hope this new hook will still be useful for some of you until then.

The plan is to do basic first level support on this list and move developer-relevant parts via cross-posting over to the old monotone-devel list. While I’m already subscribed to the new list, I encourage a couple of other developers to subscribe there as well, in case I’m not available.

I also registered monotone-users and the pre-existing one for the Debian packaging team on Gmane, but it will take a bit more time until they set them up over there, so please be patient.

Furthermore, if the “standard” receives more than 25% disapproval, it should now officially “be over” as well – if these rules would have been applied in the past, OOXML would not be an ISO certified standard as it is unfortunately today.

There are also smaller, less substantial changes. For example, the dedication to Jan van den Beld, the former head of Ecma, for his “unwavering dedication to the development and evolution of the JTC 1 procedures”, has been removed. Ironically, both Ecma and Microsoft have indeed made long-term contributions to the evolution of Fast Track in JTC1, but probably not the way they intended.

(Source)

Now the only paragraph I’m missing in the new rules obviously is a way to revoke a broken standard, but I guess this won’t happen. Lets just hope that OOXML sinks into insignificance in the next couple of years.