runaway distnoted process

Question

Sometimes I see a distnoted process suddenly spin up and chew up 100% CPU (on one core) and a ton of memory, often in the neighborhood of 1.5G or so. This happens a few times a day, starting a month or so ago.

The command line is /usr/sbin/distnoted agent, and it's started by launchd, neither of which help much. It's usually been running for somewhere between 4h and 24h before it spins up and pegs the CPU.

Web searches say distnoted manages notification delivery, and lots of other people report the same problem with it, but I haven't yet found a fix. Some people find that closing a culprit application (e.g. Skype) stops it, but I haven't found a culprit on my machine yet. I'm usually only running a few apps: Emacs (24.2 from Homebrew), Firefox, Adium, and Dash.

I'm on Mavericks on a late 2012 13" Retina MBP. Thanks in advance!

Update:

I've turned on distnoted logging in the system log by touching /var/log/do_dnserver_log, but it doesn't help much. I see lines like these (uid 501 is me, 89 I haven't found yet):

distnoted[80011]: # distnote server agent  absolute time: 48754.144787848   civil time: Wed Nov 20 10:52:03 2013   pid: 80011 uid: 501  root: no
distnoted[20]: # distnote server daemon  absolute time: 2.808112262   civil time: Tue Nov 19 09:52:24 2013   pid: 20 uid: 0  root: yes
distnoted[444]: # distnote server agent  absolute time: 16.656997509   civil time: Tue Nov 19 09:52:38 2013   pid: 444 uid: 501  root: no
distnoted[1271]: # distnote server agent  absolute time: 52.518265717   civil time: Tue Nov 19 09:53:14 2013   pid: 1271 uid: 89  root: no
distnoted[689]: Interruption - exiting now.

I've also run sudo dtruss -p PID on a spun-up distnoted process, and it spews lines like this:

kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
...

Answer 1

I've seen this too. Emacs 24.3.1, Mavericks 10.9.

I've found that the distnoted process calms down within seconds after I quit out of Emacs.

I've filed an Emacs bug here: http://permalink.gmane.org/gmane.emacs.bugs/80836

Answer 2

Summary from the OP: This was a great tool for debugging. It originally pointed me to Spotlight reindexing the filesystem, but I narrowed down the things it's allowed to index, and I still saw the problem. I ended up setting up a cron job to kill distnoted regularly. See answer farther down.

---

You can debug distnoted by creating the file /var/log/do_dnserver_log This causes the CFNotificationCenter server (distnoted) to record information about all notifications to the system log.

I would start there, reboot and look at the system log when the CPU spikes up. This should out the culprit easily.

More info on CFNotificationCenter debugging can be found in official Developer docs here: Technical Note TN2124 > CFNotificationCenter

Answer 3

I know I'm late to the party but this is a memory leak specific to Cocoa emacs on Mavericks that is fixed in the trunk. For now there is a patch you can use to build emacs 24.3 with just the fix.

https://gist.github.com/anonymous/8553178

---

Answer 4

I've been having the same problems with distnoted on El Capitan for some time. My solution isn't as harsh as killing it regularly, rather I check for it running out of control (high CPU usage), and then kill it. I use this script:

#!/bin/sh
#
# check for runaway distnoted, kill if necessary
#
PATH=/bin:/usr/bin
export PATH

ps -reo '%cpu,uid,pid,command' | 
    awk -v UID=$UID '
    /distnoted agent$/ && $1 > 100.0 && $2 == UID { 
        system("kill -9 " $3) 
    }
    '

The script is run from cron every minute with this line in crontab:

*   *  *   *  *   sh "$HOME/bin/checkdistnoted"

In practice, the script kills distnotedonce or twice a day, and typically this occurs after backupd starts.

For those not comfortable with the using the OS X shell (command line), the following script will install both the checkdistnoted script and the crontab entry:

#!/bin/sh
#
# install $HOME/bin/checkdistnoted
# setup crontab to run every minute
# 
# MWR Apr 2016
#

INSTALLCMD=bin/checkdistnoted
cd "$HOME"
[ ! -d bin ] && mkdir bin
[ -f $INSTALLCMD ] || {
    cat > $INSTALLCMD <<-"!!"
    #!/bin/sh
    #
    # check for runaway distnoted, kill if necessary
    #

    PATH=/bin:/usr/bin
    export PATH

    ps -reo '%cpu,uid,pid,command' | 
        awk -v UID=$UID '
        /distnoted agent$/ && $1 >= 100.0 && $2 == UID { 
            # kill distnoted agent with >= 100% CPU and owned by me
            system("kill -9 " $3) 
        }
        '
!!
    chmod +x $INSTALLCMD 
    echo installed $INSTALLCMD
}

INSTALLCRON="# check for runaway distnoted every minute:
* * * * * sh \"\$HOME/$INSTALLCMD\""
crontab -l | grep -q '$HOME'/$INSTALLCMD || {
    crontab -l > mycron
    echo "$INSTALLCRON" >> mycron
    crontab mycron
    rm mycron
    echo updated crontab
}

You need to save the above as install_checkdistnoted.sh on your desktop, then run Applications/Utilities/Terminal and type:

cd Desktop
sh install_checkdistnoted.sh 

If it works fully it will print confirmation of each of the steps. The script won't overwrite an existing checkdistnoted script or crontab entry.

Answer 5

i gave up and took the sledgehammer approach: kill it automatically, every minute. sigh.

i put this in ~/Library/LaunchAgents/org.snarfed.pkill_distnoted.plist:

<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.snarfed.pkill_distnoted</string>
  <key>ProgramArguments</key>
  <array>
    <string>pkill</string>
    <string>-KILL</string>
    <string>-f</string>
    <string>distnoted</string>
  </array>
  <key>StartInterval</key>
  <integer>60</integer>  <!-- every minute -->
</dict>
</plist>

and then installed it with launchctl load ~/Library/LaunchAgents/org.snarfed.pkill_distnoted.plist.

Answer 6

I've been doing different combinations of stripping customizations in order to narrow down this behavior; I think it's comint mode. On 10.9 with emacs 24.3.1 from homebrew (or from emacsforosx) the distnoted + emacs leak (they both slowly increase in memory consumption) will happen with one shell-mode buffer open. It won't if you just visit files.

Just wanted to note it here, gmane appears to be down and I keep finding this discussion on my twice weekly search for followups to this issue.

Answer 7

This seems to happen when an application somehow makes a wrong use of the notification API provided by macOS. In my case the culprit was iTerm2. After quitting it, the distnoted processes exited. Other culprits that have been identified are Emacs and iTunes.

Answer 8

I think I can only remember 2 occasions where distnoted has gone haywire. On this occasion there were 2 of them sitting to top of cpu list and one was over 400%. It happened shortly after returning to the office and plugging in a couple of external displays - one of which is usb powered - I took a guess that it might be related. I did nothing else to try and fix the problem before pulling the USB display out which brought sanity back instantly. And then plugging it back in resulted in no repeat problem.

Which proves what? No idea!

I plug them in hundreds of times and this is the first time that it occurred to me that it might be related. And since it doesn't happen everytime I plug them it, then it might have something to do with plugging them both in too quickly after each other, or something random like that. Anyhow thought I would share in case other people find it has anything to do with plugging in peripherals (if that is what an external screen is)

Answer 9

For what it's worth, I was able to fix this problem by disabling my anti-virus software.

Answer 10

This happened to me as well, distnoted was going crazy. After closing a bunch of applications, nothing helped.

Then I noticed one of those 'Report to Apple' dialogs from a crashed Python process had been left open all night.

Though it could just be coincidence, after closing the dialog the distnoted process calmed down.

Answer 11

I ran into a similar issue with distnoted a few months ago and could not track down why the CPU usage was spiking above 100%. Finally, I added an entry to my crontab to killall distnoted every 2 minutes which solved my problem.

Recently, I have been having an issue with Sublime Text where typing subl path/to/file was failing to open the file correctly in the Sublime Editor. A restart of the app fixed the issue, but it quickly began to happen again.

After racking my brain to no end, I identified the fact that I was killing distnoted process every 2 minutes to why the subl command had mysteriously stopped working.

The conclusion: the super high CPU usage may have been related to sublime. Now that sublime has updated, hopefully my conclusion is correct, CPU usage remains low, and my subl command returns to working as expected now that distnoted is running again without my crontab killing the process every 2 minutes.

Answer 12

I've had this problem too, for quite some time now, but intermittently. Apparently distnoted is part of iTunes and has caused problems on Windows as well. When I killed iTunes (which was playing a song), the distonted process that was using 400% of my CPU (I have 4 cores) stopped being a problem.

So my answer, until I know better, is to recommend that you kill iTunes, not distnoted, and let us know what happens.

Answer 13

I also see distnoted go haywire, in my case it seems related to fontd. I have three distnoted running, one for _spotlight, one for _distnote and one for my user.

distnoted   0,0 6:39,85 2   0   101 _distnote   0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit
distnoted   0,0 0,05    2   0   642 _spotlight  0 bytes 0 bytes     Yes     -   No  No  No  0 bytes 0 bytes 64 bit
distnoted   82,1    1:19:38,30  49  1   353 nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit

Whenever distnoted eats cpu (30-90%), fontworker and fontd eats about 30-60% cpu each. As soon as I kill fontd, distnoted and fontworker for my user calms down. Killing fontworker does nothing. After a couple of minutes when fontd has restarted and been running a while it all starts again.

fontworker  27,2    52,81   4   1   1073    nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit

fontd   32,6    1:07,41 6   0   1072    nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit

I have no clue on why this is happening…

Answer 14

Peter Buckley is right, I'm wrong. I hate it when that happens.

Don't remove distnoted, the next boot will be no fun at all.

wrong> I took a more sledgehammer approach
wrong> 
wrong>    sudo mv /usr/sbin/distnoted /usr/bin/distnoted.unwanted
wrong>
wrong> This is a work machine and I have no interest in sync'ing with iTunes.