by Luc Saffre · in Hosting (system administration) · assigned to Luc Saffre

Summary

Children

• #6768 (How to configure our mail servers) (by Luc Saffre in Hosting (system administration) assigned to Luc Saffre)

Comments

• 5 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : But the problem is fixed now: luc.lino-framework.org/blog/2026/0510.html
• 5 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : NB I am still unable to send emails to the mailinglist. I sent an email to the list-admin, but they too didn't respond. On the other hand the list members are really great! They are discussing among them self to explore my problem. Here is the post that is very probably my solution: lists.debian.org...
• 5 months, 3 weeks ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : First answer came half an hour later: lists.debian.org/debian-user/2026/05/msg00019.html
• 5 months, 3 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : I finally finished my problem description and sent it to the debian-user mailing list.
• 5 months, 3 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Read the docs about maildrop here.
• 5 months, 3 weeks ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : But there is only one directory named maildrop, and it is under /var/spool/postfix, so it should not be read-only. And btw it is empty.
• 5 months, 3 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : This might be caused by systemd’s protection features, which are enabled for logrotate and Postfix. In particular, ProtectSystem, if set to “full” or “strict”, will result in /etc being read-only (found here).
• 5 months, 3 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : No, the problem is *not* solved. I thought that it is related to the postfix config, but the problem persists even when sending mail to root works.
• 5 months, 3 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I closed this ticket because the original problem is solved. And I started a new ticket #6768 (How to configure our mail servers)
• 5 months, 4 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I subscribed to the debian-user mailinglist and plan to describe my problem there (I checked the "digest" option because this list has more than 10 mails per day...).
• 5 months, 4 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Today (once more) I worked exclusively on this problem. I know now that /etc/monit/conf.d/lino.conf is the culprit, but I still don't understand why it's failing and how it manages to cause such a trouble. I wrote a blog entry about my problem: luc.lino-framework.org/blog/2026/0427.html
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Yes, the lino.conf for monit, generated by getlino, is the culprit. This is also the explanation why the same problem survived even a rebuild of the server.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : It is possible that monit (i.e. our monit config file made by getlino) is the culprit! I removed the file and the warnings didn't come back after reboot.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I started to dig into the Debian documentation because that's the place where it would make sense to contribute (while most other platforms are not free).
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Debian switches to read-only file system when it detects corruption, in order to prevent further corruption. So I ran "fsck -fn". It gives some error messages but they seem rather harmless: (main) luc@lumi:~$ sudo fsck -fn
  fsck from util-linux 2.41
  e2fsck 1.47.2 (1-Jan-2025)
  Warning! /dev/sda1 is moun...
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : And the warnings "mail_queue_enter: create file maildrop/231549.2196: Read-only file system" are still there. The first one comes quickly after rebooting. They come every 10 seconds (I guess that postdrop is configured to run every 10 seconds). In the beginning, after rebooting, it's only one warnin...
• 6 months ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : The sudo warning seems to come from unattended-upgrades. Seems unrelated. I said "unattended-upgrades --dry-run --debug" and it seems to work fine.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : Here is a list of all yellow and red warnings in journalctl-b: apr 25 05:00:01 lumi kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied!
  apr 25 05:00:01 lumi kernel: Speculative Return Stack Overflow: WARNING: See kernel.org/doc/html/latest/admin-guide/hw-v>;
  apr 25 0...
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : The first warnings in journalctl -b were from OpenDKIM. We don't need this any more since we send via the zone relay host. So I said "apt purge opendkim".
• 6 months ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Oops, no, they are still there. (I forgot to sudo when saying journalctl).
• 6 months ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Now after rebooting, I no longer see the "Read-only file system" warnings.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (2 replies) : I read this and changed two more postfix settings: # Accept relay requests only from localhost (IPv4)
  mynetworks = 127.0.0.0/8
  # Reject relay from anything outside mynetworks
  smtpd_relay_restrictions = permit_mynetworks, reject
  
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : And even after rebooting it continues to say "postfix/postdrop[2117]: warning: mail_queue_enter: create file maildrop/545833.2117: Read-only file system"
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I set mydestination to empty in the postfix main.cf on lumi.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I added PasswordAuthentication no to /etc/ssh/sshd_config
  
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Even when running with the mail service, the phenomen is back. postfix/postdrop[7430]: says "warning: mail_queue_enter: create file maildrop/700470.7430: Read-only file system"
  
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I gave up. I ordered mail service from zone and removed postfix and dovecot from lumi.
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : OMG and the problem is back once more. This time the slowness is caused by 259 postdrop processes, 259 sendmail processes and 261 sudo processes. "ps -e -orss=,args= |awk '{print $1 " " $2 }'| awk '{tot[$2]+=$1;count[$2]++} END {for (i in tot) {print tot[i],i,count[i]}}' | sort -n" says:
  18376 /usr/...
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : There was at least one missing entry in the /etc/opendkim.conf: Socket local:/var/spool/postfix/opendkim/opendkim.sock
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : There is still a memory problem, but with a new cause. Some postfix config error. It is saying lots of warnings "connect to Milter service local:opendkim/opendkim.sock: No such file or directory".
• 6 months ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : For the Postfix and Dovecot config I created a separate ticket #6753 (Install Postfix on lumi (Debian 13)).
• 6 months ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : Last thing to restore is postfix and dovecot...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I probably fixed a subtle bug in getlino: it asked for the site_domain but then ignored your answer. When installing ets with the site_domain lino.laudate.ee, it ignored my answer, leading to nginx config file for laudate.ee, and then certbot updated the wrong nginx config file. I think i fixed this...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : Wow, this was a long session. But basically everything went well, the laudate server is now reincarnated under its new name lumi. The ets site still has a certbot problem (getlino is so buggy!), sinod is not yet back, postfix and dovecot aren't yet configured.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : When I create a new server on laudate, I also want to change the hostname to lumi because laudate is the name of a repository that happens to be published on lumi as laudate.ee, but the server also hosts other sites.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I made a copy of the ets site on my notebook and got it up and running (I wanted to see the data on my notebook before deleting the laute server because i have been sitting quite some hours on this content). I copied /etc/nginx and /etc/supervisor to ~/backup on my notebook. I think that now I'm read...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I start to think that maybe i should simply let Hetzner reformat the virtual machine and install a virgin new Debian.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I read a docs page by Hetzner restarted laudate in rescue mode and then tried to run fsck: root@rescue ~/.oldroot/nfs # parted /dev/sda 'print'
  Model: QEMU QEMU HARDDISK (scsi)
  Disk /dev/sda: 41.0GB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt
  Disk Flags:
  Number Start End Size ...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Maybe the file system is really broken?
  # fsck -n -f fsck from util-linux 2.41
  e2fsck 1.47.2 (1-Jan-2025)
  Warning! /dev/sda1 is mounted.
  Warning: skipping journal recovery because doing a read-only filesystem check.
  Pass 1: Checking inodes, blocks, and sizes
  Pass 2: Checking directory structure
  Pass 3:...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Note that it "Read-only file system", not "permission denied". "postfix check" doesn't report any issue.
• 6 months, 1 week ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I rebooted and these warnings indeed have disappeared from the log. But the warnings dropmail failures continue to reappear, the first yellow message is now "10:03:53 laudate postfix/postdrop[1177]: warning: mail_queue_enter: create file maildrop/591771.1177: Read-only file system"
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I found this thread, which suggests to simply delete these three files in /etc/init.d. Hm...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (1 reply) : Desperate as I am, I read this discussion and rebooted the server three times in a row. I can watch what happened after the last boot with "journalctl -S 2026-04-18T06:54:05". The first yellow log entry is at 06:54:16 for unit systemd-sysv-generator[413]: SysV service '/etc/init.d/sendmail' lacks a n...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Omg, it's locked again. So the some lacking restrictions (smtpd_recipient_restrictions, smtpd_helo_restrictions..) were not the cause.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I also reinstalled dovecot. Now Thunderbird can send from sr to laudate, but not send via laudate.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I compared the postfix main.cf file with the one on LF and did the following changes on laudate: I deactivated relay_domains = $myorigin, as well as
  mailman_destination_recipient_limit = 1
  cyrus_sasl_config_path = /etc/postfix/sasl
  I activated: smtpd_recipient_restrictions and smtpd_helo_restrictions...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Once more the lock is back, And dovecot is not installed. And journalctl reports thousands of lines apr 16 09:21:38 laudate postfix/postdrop[27873]: warning: mail_queue_enter: create file maildrop/510779.27873: Read-only file system
  
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : The lock was back. So dovecot is again the main suspect. I removed it again.
• 6 months, 1 week ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I saw that /etc/cron.d still contained three files from our experiments with mailman3 in 2018. I disabled them: # cd /etc/cron.d
  # mkdir disabled
  # mv mailman* disabled/ And then rebooted the system. Now let's see whether the problem comes back!
• 6 months, 1 week ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I said "sudo killall postdrop" and the lock went away, the server behaved normally again.
• 6 months, 1 week ago by Luc Saffre in reply to Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I optimized this approach and said "sudo journalctl --no-pager -S '2026-04-14 23:00:00'". This gave me interesting information
  apr 15 02:42:26 laudate kernel: Out of memory: Killed process 27265 (python) total-vm:302128kB, anon-rss:163784kB, file-rss:124kB, shmem-rss:0kB, UID:0 pgtables:492kB oom_...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) (3 replies) : The server is stuck once more, but this time I didn't power it off, I said "ssh laudate" and then waited patiently until it let me in. After a while it did so, and it behaved normally for a minute. I guess that I was in a valley of the canyon, the CPU usage graph at Hetzner looks like a canyon, the ...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : It seems to be a bug that happens when RAM is getting low. I saw several threads about it, many of which were years old. cat /proc/sys/vm/swappiness and cat /proc/sys/vm/vfs_cache_pressure show their usual default values 60 and 100. Should I fiddle with these? I can't believe that. Then some people r...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : But after a while the kswapd0 process was back (with same PID 53). And Hetzner web interface shows a constant 200% CPU usage.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I did "ssh laudate" and noticed that it is very slow. top showed that systemd and kswapd0 are both using 60..80% of CPU. I read this thread and tried "kill -9 53" (53 was the PID of kswapd0) but that didn't help. Then i tried "killall -9 kswapd0" and it helped.
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I moved the journal-vacuum from daily to weekly in order to test whether this is the culprit. Though I'm quite sure that it is innocent because daily cron jobs didn't yet run, IOW I must continue to search for the possible reason. Next suspect is dovecot: maybe some external agent manages to sign in...
• 6 months, 1 week ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Situation similar to yesterday. web server and ssh time out. I powered the server off, as yesterday.
  
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : idea: Is is possible that my journal-vacuum.sh daily cron job triggered this?"
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : There it is back, as if nothing had happened. There is enough disk space, df -h says: Filesystem Size Used Avail Use% Mounted on
  /dev/sda1 38G 28G 8,8G 76% /
  
  
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Shutdown didn't seem to do anything. I now asked it to power off. "Are you sure you want to power off this server? This will do a hard shutdown of your server, the same as pulling a power cord. This action may cause data loss."
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : I used the Hetzer web interface to shutdown the server. They asked "Are you sure you want to shutdown this server?This will send an ACPI signal to your server. If your server is using a standard configuration, it will do a soft shutdown. If this does not work, this will lead to a hard shutdown of yo...
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) :
• 6 months, 2 weeks ago by Luc Saffre about #6733 (laudate server not responding and kswapd0 CPU usage at 200%) : Created new Ticket #6733 (laudate server is offline?).