Bad Ideas: Cron Replacement on OSX 10.11

2015-10-02 23:24 EDT

OSX 10.11, aka El Capitan, was released a few days ago and introduced System Integrity Protection (SIP). SIP is a security mechanism which, at its most basic, prevents the root user from performing all sorts of actions. It's conceptually very similar to SELinux but with none of the configuration ability. For the purposes of this post, the key "issue" is that SIP prevents root from writing to protected directories, namely /usr (amongst others).

In setting up a new machine tonight, I ran into an issue. I was unable to create a new crontab. After some digging, I realized that all of cron's magic is in /usr/lib on OSX. SIP is guarding /usr now so there's no way to get a new crontab going. That's … problematic.

Years ago, Apple transitioned all of its tooling away from cron into launchd. Cron-like functions are configured via plist files in ~/Library/LaunchAgents or /Library/LaunchAgents (for apps to be executed as root). The syntax is clunky as all hell but the system works. The "right way" would be to rewrite my desired crontab entries into individual launchd plists. That sounds really boring. Besides, why do things the "right" way when I could do something terrible instead?

I decided to build a replacement for cron via launchd and some shell. It ended up working like /etc/cron.hourly and /etc/cron.daily that anacron usually provides.

Here's the launchd file for the 5 minute execution. This lives in ~/Library/LaunchAgents/com.sungo.fivemincron.plist. Load it up by running launchctl load com.sungo.fivemincron.plist. If you are a tmux user, you must run this command outside of tmux or you'll get a permission denied error.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">







Hopefully it's obvious but that runs /Users/sungo/bin/five_minute_cron every 300 seconds. For a normal person, you can probably stop here. Put whatever code you want to run in-series in that script and you're good.

I'm not normal.

With cron, if I set several commands to run every five minutes, they run in parallel. Putting those commands into a script gets them to run in series which is not what I want.

gnu-parallel to the rescue.

Here's what five_minute_cron looks like with a dash of crazy.

#!/usr/local/bin/parallel --shebang --will-cite :::


-shebang tells parallel that the following lines make up its config file. -will-cite gets rid of the really fucking annoying plea for citations. (Yeah, I don't even, either.) The ::: tells parallel to use the remaining arguments as command input source rather than waiting for commands via stdin.

All in all, that script runs the commands listed in parallel. Launchd will run the script every 300 seconds.

So now I have a cron-ish replacement using launchd and gnu parallel.

Surviving A Botswarm - A User's Guide

2014-07-18 21:58 EDT

So yeah. has been under a sustained bot incursion for a few days now. The network opers are actively working the problem the best we can. At the end of the day, though, with the existence of cheap instant virtual machines and the wonderful world of Tor, this sort of incident is difficult to stop completely.

(For the record, I don't view this as an attack. Near as I can tell, this is one kid getting his jollies by occasionally spamming channels. This thing is mostly automated except when he gets bored and tries to taunt me with rainbow colors. Which, incidentally, is why hereafter this person will be known as The Rainbow Warrior.)

Anyway, I don't really want to talk about bored 14 yr olds.

Let's talk instead about what you can do to survive a botswarm.

First, set your client to ignore joins, parts, quits, and maybe modes. In irssi, that command is "/ignore * JOINS PARTS QUITS MODES". I have this set in my client in most channels, particularly the quiet ones. It cuts the noise significantly.

Some clients provide a function to ignore nick changes as well. In irssi, add NICKS to that command above. I don't recommend this in normal use but right now, with bots that change nicks constantly, it might be a good idea.

Second, figure out how to make your irc client strip color codes. If you're an adult hanging out with adults, chances are you forgot you could do color in irc. It's safe to strip out and unless you're making a foray to the wonderful land of Dalnet, you'll never miss it. (I'd give you the irssi command but I don't know it yet. If you know, clue me in and I'll update this.)

Third, ignore CTCP. (Add CTCPS to the ignore command listed above.) This is 'client-to-client protocol', again something you probably forgot about. It's mostly used to negotiate DCC sessions and annoy people via the 'notice' message type.

Side note: You really should ignore DCC as well, just from a security perspective. In this day and age, it is a ridiculously bad idea to allow a direct connection to your irc client from another client on the internet. Historically, DCC was used to transmit files. You'd join #warez on dalnet, see a bot's message (in color, no doubt) and use DCC to download files from it. I have no idea why we did this instead of usenet but whatevs. This was always a bad idea. Add "DCC" to that ignore command I keep referencing to ignore this blast from the past too.

As a user, that's really all you can do, sadly. Ignore the junk as best you can. If it drives you nuts, it's ok if you walk away for a while. We understand. I'm on the response team for this shit and I often slam the laptop lid closed and walk away for a while.

Let's talk about what channel operators can do to help protect their users.

First, go read this New IRC Channel Operator's Guide from Even if you're not a new operator, go read it anyway.

If you suddenly get hit by a flood of joins from people you don't know, just make the channel +im temporarily so they can't keep coming in and can't flood in the channel. Note they can still cause flooding such as by rapidly changing their nicknames. Now just kick them without bans since they cannot rejoin while you are +i, that gives you time to set proper bans after you've kicked them all out.

There are three basic defensive postures. (These can be aided by the use of control bots but do not require it.)

  • Set the channel +k. This "keys" the channel and requires the user to know a special string to join. Only give that key to people you like and if the bots get in, you know one of the people you like is a jackass.

  • Set the channel to +i. This sets the channel to "invite only". Users must be allowed in via the /invite command to access. On, users can use the /knock command to request an invite. If you do this, I suggest leaving the channel -s and put this info in the topic. That way, a user can do '/list #yourchannel' and see that they need to /knock first.

  • Set the channel to +m. This moderates the channel and only people with +o (ops), +h (halfops) or +v (voice) can speak. People can join but won't be able to speak until given +v. If someone, bot or not, gets out of control, you can just -v them. It's pretty much the same as "hellbanning", to use the modern parlance. Many IRC clients can be configured to automatically give +v to people when they join. (That is typically much easier and a better idea than dealing with channel control bots.)

  • Make heavy use of halfops. The +h (halfops) user mode is one of the most underutilized feature of, in my opinion. Halfops have the ability to do all the normal ops stuff. Kick people, ban people, voice people, etc. They cannot however perform those actions against anyone who is +o. It's a really nice way to create a class of users who can help police the channel but are visibly different from the channel leadership.

That's all I have for now. If you find any tips I should know about or blog about, grab me on First Community Opers

2014-05-31 15:14 EDT

It is my privilege to announce the first batch of community opers. They are tasked with drafting a charter and maintaining the peace on under the terms of the Standards Of Conduct. We'll have our first meeting at YAPC::NA 2014.

  • Coordinating Network Opers: sungo (me), mst (Matt Trout)
  • Supervising Network Oper: perigrin (Chris Prather)
  • Community Opers (in alpha order):
    • ether (Karen Etheridge)
    • garu (Breno G. de Oliveira)
    • genehack (John Anderson)

Sometimes, when governance is put in place and when the leaders are announced, the shady parts of the community come forward to harass them. Let me be really clear about this. Folks, you had plenty of time to discuss governance sensibly. If you have issues with these community opers, we can discuss it. But if you do more than discuss it, if you harass them in any way, I will take it as a threat to network safety and security and I will respond as strongly as I am allowed under the SoC and governance rules.

It's really unfortunate that I feel the need to document my stance but we've dealt with this before and I lost my sense of humor about it a long time ago. Governance

2014-05-31 14:49 EDT

After a much longer public review period than I intended, the governance proposal is complete. There was no grand explosion, no riots at my door, no death threats. I'm willing to call that a success and treat the governance proposal as accepted.

What follows below is the governance document in its final form. It is no longer a proposal but the rule of law. (That's a bit more dramatic sounding than it probably should be. But you get the point.)

In a little while, I'll announce the first set of community opers. The infrastructure to support the new opers as documented below is not up and running yet but will be in the very near future. Hopefully before our first face-to-face at YAPC::NA 2014.

Which reminds me, as specified below, we'll be having an governance BOF at YAPC. It is open to all. Date/time are unknown at this time and I'll blog when I know more.

Governance of

Community Opers

A channel, #magnet-srb, will be created, including a logging bot. These logs will be completely public and linked from the website. This channel will serve as a open review board for Standards of Conduct (SoC) violations, including oper misconduct.

The SoC Review Board (SRB) will be staffed by a new class of oper, the community opers, who will be shepherded by Chris Prather (perigrin), an existing network oper. Community opers will have full authority over all channels and will be tasked with enforcing the SoC. Their rulings on community matters will serve as rule of law for the network. No changes will be made to the official standards of conduct without consulting the SRB.

The SRB must consider individual channel policy as well as the regular SoC in their rulings. As noted in the SoC, channel ops can create local policy, provided this policy is documented and made available to new channel members. The SRB can at any time determine that local channel policy is in direct conflict with network policy. The channel in question is expected to alter their policy or leave the network.

If a user thinks the policy of a public channel conflicts with network policy, they may bring the policy to the SRB for mediation. This is to be an approach of last resort and the user must provide evidence of channel discussion about the policy and detail why they think the SRB should intervene. The SRB may decline to intervene and, if so, the channel policy remains intact. Unless the channel constitutes a grave threat to the community or network, the policies of any +s, +p or +k channel will not be considered by the SRB.

If the SRB is notified of or discovers a channel or person who constitutes grave harm to the network, they must report this to the network opers who will act to purge the offending parties and take legal action if necessary. A great and sadly common example is the presence of a botnet command and control channel.

Power Limitations

Community opers will not have voting rights for or be consulted about network infrastructure or network security. It is expected, however, that they will be kept apprised of any ongoing security issues or outages.

Community opers will not have the ability to issue klines or glines, as these are a function of network operations. If the SRB determines that a kline or gline is necessary as part of SoC enforcment, they will bring this recommendation to the network opers via Chris Prather (perigrin), sungo or Matt Trout (mst). It is expected that the network opers will act on the recommendation of the community opers but they may challenge the decision if they wish. The matter will then be discussed openly in #magnet-srb.

Oper Oversight

Any oper action, including that of the community opers, can be challenged before the SRB. It is the responsibility of the SRB to determine if an oper has acted in bad faith. If misconduct has occurred, the SRB will designate someone to discuss the matter with the oper in question privately. Resolution of that discussion will be reported back to the SRB. The SRB has the power to revert channel bans, op status, etc and its ruling will be final.

Should the oper respond negatively or repeatedly act in bad faith, their status as oper must be evaluated. In the case of a community oper, the SRB can order that their oper status be removed. This matter must be brought to the community oper coordinator directly who will carry out that order in coordination with the network opers.

An unfortunate exception exists for network opers, who cannot have their status removed due to their operation of an irc node. Due to this special nature, when a network oper violates policy, the other network opers will deal with the individual privately and in a manner commensurate with the infraction. The coordinator of the Community Oper team will be involved in this process.


An initial group of three community opers, in addition to the aforementioned shepherds, will be hand selected by the current network opers. Initially, they will be given two tasks. First, they shall work with the network opers to draw up an official charter based on the rules and guidelines laid forth in the proposal. Second, they shall draft policies regarding the selection and removal of future community opers.

A initial face to face meeting will be held at YAPC 2014. Within 6 months of that gathering, the SRB must publish these policies on the website and select at least two additional community opers.


Current unwritten network policy states that server admins can create opers as needed for the maintenance of their node(s). As such, community opers, for now, will connect to a private node run by sungo and co-maintained by mst and perigrin. They will be given special hostmasks (something like 'community.oper') to make their status more visible and to help disguise their connection parameters. Both the private node and hostmasks will help protect the SRB from network attacks that may occur as a result of their governance.

It is expected that a SPOF-free architecture for community oper infrastructure will be created within one year of the ratification of the SRB selection policies.

The Changes

2014-04-14 20:35 EDT

In the last few weeks, I've made some changes to (Yes, I said "I". I'll explain in a bit.) The response has been mixed. Several "yay!"s, several "wat"s and a silence I'll assume is "meh". I want to take a bit and put some context around these changes and why I'm making them.

Let's start with some history. wasn't born It was born in 1999-2000 as MAGnet or Rhizomatic, depending on who you ask. It was a refuge from EFnet #perl which had gotten a bit too, I guess, n00b-filled for everyone. So, a splinter group built an irc server and thus Rhizomatic was born. (I prefer Rhizomatic for the philosophical roots.) There were maybe a dozen or two users on a single server. Really, there was a single channel, too: #perl. Just as lawless as EFnet #perl but a more curated community. Curation by obscurity.

Over time, we added another server to cover downtime, then a third. Oper blocks were handed out to people we liked. There were no real rules because it was a network of friends.

Then it all changed.

One day, we woke up to discover that the domain had been pointed to our little network. Suddenly, we had n00bs and DDoS and botnets. Suddenly, we had a need for policy.

We purged the oper blocks down to just the people that run the network and drafted the most simple of policies. "We, the opers, run the network. Don't fuck with us." It was simple, elegant, and reflected our general annoyance at being bothered. But we also had an internal rule. No one gives a shit about the channels. We ran the network not the channels. We restored ops when necessary and that was pretty much it. We were still a network of only a few hundred. is a very different place now. Years have passed. We've grown from a few hundred to 1400 strong. We've grown from one main channel to 460 channels, most of which are community channels dedicated to a particular open source project. We've gone through half a dozen different node providers and even more opers.

All this time, though, general policy has stayed the same. Don't fuck with the opers.

Let's go back to that "I" from the beginning. Over the years, the oper group has shrunk for various reasons. I am the only surviving oper of the first few years. There are three of us who take care of the day to day, really. When it comes to policy, I'm the one deciding changes and making them happen. I control the server deployments. When I decided to make changes, I didn't need to consult anyone. (I did, though.) This is the definition of SPOF and the tendency for policy-developed-by-navel-gazing is too high.

You deserve better.

The new Standard of Conduct and the proposed governance structure are designed to two things. First, they codify what we've done informally for a long time. Second, they give control of the community back to the community.

Our Standard of Conduct really is, fundamentally, the Reasonable Person Principle spelled out in long form. There's nothing new in there, from my perspective. Don't be an asshole. In case you're not sure what that means, we helpfully spell it out.

The governance proposal makes official a program I started a year or two ago. To deal with the vast segments of the community I'm not aware of, I asked a couple of people to be opers so they could take care of the needs of their community segment. This proposal makes that official and puts them in one channel and, really, gives them a single goal. "Make this irc network into the best it can be for the community it serves."

Honestly, I do not expect this to change your day to day life on Well, unless you're an abusive asshole in a public channel. Then, yeah, you might want to re-examine your life choices. Otherwise, nothing in your irc life changes and, if there's a problem, you know who you can talk to about it.

And, yes, it is still just a proposal. If we decide this proposal a bad idea, as a community, we'll figure something else out.

From a network ops perspective, nothing is changing. Still the same ol' curmudgeon assholes keeping the lights on. You'll see us when the monsters come out to play.

I do want to be very clear about something, though. Part of the goal of the new SoC and the new governance is transparency. I will not, however, be transparent about the server architecture or day to day operations of the network. There's secret sauce in the system that helps mitigate and prevent the DDoS issues we had in 2010. If it affects the userbase, I'll be as transparent as I can but some things will always remain private.

There are a few other things brewing. Nothing that really affects anyone, though. I promised a way that you can donate to the server operations budget and that's still coming. I'm also working on some other ways people can pitch in.

For now though, I really need everyone to provide constructive feedback on the governance proposal. Please comment on the gist, grab me on irc or twitter.