CPU Wait issue caused somehow probably by MySQL and possibly related to max_allowed_packet issue

cpu wait is ridiculously high with nothing but mysql running (with the server actually serving, it was 90+% consistently).

top - 05:52:29 up 3 min, 1 user, load average: 0.84, 0.53, 0.22
Tasks: 69 total, 1 running, 68 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.4%us, 1.1%sy, 0.0%ni, 47.8%id, 46.5%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 1035172k total, 1019344k used, 15828k free, 78404k buffers
Swap: 2650684k total, 552k used, 2650132k free, 712048k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2941 mysql 20 0 274m 97m 4804 S 22 9.7 0:10.24 mysqld
2942 root 20 0 3004 616 540 S 2 0.1 0:00.06 logger
3000 ben 20 0 2376 992 804 R 1 0.1 0:00.06 top
1 root 20 0 2100 700 596 S 0 0.1 0:01.22 init

Here's top at the higher load, MySQL and Apache running:

top - 02:31:22 up 11 min, 2 users, load average: 12.77, 13.11, 7.45
Tasks: 171 total, 2 running, 169 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.2%us, 1.5%sy, 0.0%ni, 0.0%id, 93.2%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 1035172k total, 1021108k used, 14064k free, 7128k buffers
Swap: 2650684k total, 1071536k used, 1579148k free, 194148k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2114 mysql 20 0 267m 100m 2876 S 12 10.0 1:11.85 mysqld
2758 www-data 20 0 115m 33m 13m D 3 3.3 0:05.18 apache2
2937 www-data 20 0 105m 19m 8636 D 3 1.9 0:00.68 apache2
180 root 15 -5 0 0 0 S 1 0.0 0:06.76 kswapd0
2985 www-data 20 0 99.0m 7936 3412 D 1 0.8 0:00.08 apache2
2987 www-data 20 0 98.2m 7104 3480 S 1 0.7 0:00.04 apache2
2914 www-data 20 0 115m 35m 14m S 1 3.5 0:03.10 apache2
2970 www-data 20 0 98.7m 7816 3448 D 1 0.8 0:00.10 apache2
2986 www-data 20 0 98.2m 7028 3344 S 1 0.7 0:00.02 apache2
1 root 20 0 2100 540 512 S 0 0.1 0:01.26 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration

Stopping apache and stopping mysql and starting mysql and doing this is what i think fixed it:

mysqlcheck -u root -p --auto-repair --check --optimize --all-databases

http://blog.taragana.com/index.php/archive/mysql-tip-how-to-check-repair-optimize-all-tables-in-all-databases/

Note that a couple server reboots did not fix it.

The constant max_allowed_packet error in the syslog, and one odd e-mail one:

Mar 3 06:43:56 server1 apache2: PHP Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (0, 'php', '<em>Got a packet bigger than &amp;#039;max_allowed_packet&amp;#039; bytes\nquery: UPDATE cache_views SET data = &amp;#039;a:4:{s:6:\&amp;quot;tables\&amp;quot;;a:138:{s:4:\&amp;quot;book\&amp;quot;;a:5:{s:4:\&amp;quot;name\&amp;quot;;s:4
Mar 3 06:43:57 server1 postfix/pickup[11830]: 85DE4944D94: uid=33 from=
Mar 3 06:43:57 server1 postfix/cleanup[11920]: 85DE4944D94: message-id=20100303124357.85DE4944D94@server1.agaric.com
Mar 3 06:43:57 server1 postfix/qmgr[2621]: 85DE4944D94: from=www-data@server1.agaric.com, size=1877, nrcpt=1 (queue active)
Mar 3 06:43:59 server1 postfix/smtp[11922]: 85DE4944D94: to=se.ma.u.tkin.g.v.a.nf.u@gmail.com, relay=gmail-smtp-in.l.google.com[209.85.222.94]:25, delay=1.9, delays=0.12/0.02/0.35/1.5, dsn=2.0.0, status=sent (250 2.0.0 OK 1267620239 34si12014557pzk.119)
Mar 3 06:43:59 server1 postfix/qmgr[2621]: 85DE4944D94: removed

This is not syslog, then, but legacy Drupal 5 sites with database logging, these errors are showing up from the PHP error caused when Drupal cannot log the MySQL max_allowed_packet error (by, in fact, triggering the same error).

Mar 3 06:45:26 server1 apache2: PHP Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (0, 'php', '<em>Got a packet bigger than &amp;#039;max_allowed_packet&amp;#039; bytes\nquery: UPDATE cache_views SET data = &amp;#039;a:4:{s:6:\&amp;quot;tables\&amp;quot;;a:138:{s:4:\&amp;quot;book\&amp;quot;;a:5:{s:4:\&amp;quot;name\&amp;quot;;s:4

Weirdness?

Mar 3 07:15:01 server1 postfix/pickup[11830]: CF909944D94: uid=1048 from=
Mar 3 07:15:01 server1 postfix/cleanup[12361]: CF909944D94: message-id=20100303131501.CF909944D94@server1.agaric.com
Mar 3 07:15:01 server1 postfix/qmgr[2621]: CF909944D94: from=anjali@server1.agaric.com, size=628, nrcpt=1 (queue active)
Mar 3 07:15:01 server1 postfix/local[12390]: CF909944D94: to=anjali@server1.agaric.com, orig_to=, relay=local, delay=0.11, delays=0.05/0.01/0/0.06, dsn=2.0.0, status=sent (delivered to mailbox)
Mar 3 07:15:01 server1 postfix/qmgr[2621]: CF909944D94: removed
Mar 3 07:15:02 server1 postfix/qmgr[2621]: 9A96AABC041: from=www-data@server1.agaric.com, size=1296, nrcpt=1 (queue active)
Mar 3 07:15:03 server1 postfix/qmgr[2621]: 8354C944D8D: from=www-data@server1.agaric.com, size=1131, nrcpt=1 (queue active)
Mar 3 07:15:03 server1 postfix/smtp[12397]: connect to mail.mymail-in.net[217.20.163.8]:25: Connection refused
Mar 3 07:15:06 server1 postfix/smtp[12363]: connect to stylebrand.com[208.73.210.27]:25: Connection refused

Mar 3 07:15:06 server1 postfix/smtp[12363]: 9A96AABC041: to=xxxxxxx@stylebrand.com, relay=none, delay=130699, delays=130695/0.02/3.1/0, dsn=4.4.1, status=deferred (connect to stylebrand.com[208.73.210.27]:25: Connection refused)
Mar 3 07:15:33 server1 postfix/smtp[12397]: connect to relay.mymail-in.net[217.20.163.69]:25: Connection timed out
Mar 3 07:15:33 server1 postfix/smtp[12397]: 8354C944D8D: to=traweller@mymail-in.net, relay=none, delay=147435, delays=147404/0.01/30/0, dsn=4.4.1, status=deferred (connect to relay.mymail-in.net[217.20.163.69]:25: Connection timed out)

How to increase max allowed packet size:

http://www.liferay.com/web/guest/community/forums/-/message_boards/message/124969

Syslog instead of database logging

Problem is knowing which site is making the log.

http://support.bryght.com/vps/syslog suggests giving each site a different "local" and directing each to its own logfile.

but that limits us to seven sites on a server!!
admin/settings/logging/syslog