#5588 (Jane is not making daily snapshots)

by Luc Saffre · in Hosting (system administration) · assigned to hannes

Kokkuvõte

Children

#5591 (Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this me>) (by Luc Saffre in Hosting (system administration) assigned to Luc Saffre)
#5592 (Server 'unix_http_server' running without any HTTP authentication checking) (by Luc Saffre in Hosting (system administration) assigned to Luc Saffre)

Kommentaarid

2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Now Jun 01 11:49:59 laudate.ee supervisord[3571345]: 2024-06-01 11:49:59,795 INFO spawned: 'libreoffice' with pid 3571355
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Seems i solved the issue. In /etc/supervisor/conf.d there is libreoffice.conf. I replaced line: command = libreoffice --accept=socket,host=127.0.0.1,port=8100;urp; --nologo --headless --nofirststartwizard
by line
command = /usr/lib/libreoffice/program/soffice.bin --accept="socket,host=127.0.0.1,por...
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : This doesn't work, cronjob alert came again. If i check ps aux | grep libre I see: root 3543428 0.0 1.2 416884 25012 ? Sl May30 0:00 /usr/lib/libreoffice/program/soffice.bin --accept=socket,host=127.0.0.1,port=8100;urp; --nologo --headless --nofirststartwizard I see that libreoffice serv...
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : In /etc/cron.hourly of jane modified healthcheck accordingly: chmod go+x healthcheck chown -c root:www-data healthcheck restarted cron and supervisor.
2 aastat, 4 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : @hannes, I reviewed the page [url hosting.lino-framework.org/oood/ More about the LibreOffice service] in the hope that it helps you.
2 aastat, 4 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : There is now still that hourly error message "libreoffice is not running." @hannes what do you know about it?
2 aastat, 4 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : I made the [url gitlab.com/lino-framework/getlino/-/commit/82d4b7521dc0c6d2422c7ab7f14f5bc215fc735f change to getlino].
2 aastat, 4 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Ahaa! Finally we got a reasonable explanation!
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : In jane, /etc/cron.daily I renamed make_snapshot_jane.sh to make_snapshot_jane and make_snapshot_ets.sh to make_snapshot_ets, restarted cron and today morning the backup was done. Seems cron doesn't like file extensions. Indeed, from stackoverflow.com/questions/16218705/crontab-not-executing-bash-s...
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Gave a try to configuration where the two files are in/etc/cron.daily directory, but without extension .sh.
2 aastat, 4 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Restored configuration, where 2 .sh files are in /usr/local/bin and cron.daily runsa single file /etc/cron.daily/linobackups.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : The versions of cron for avanti and jane are the same.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Are the cron versions different?
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : In avanti, /etc/cron.daily we have: -rwxr-xr-x 1 root www-data 153 Mar 27 2023 make_snapshot_prod.sh and we have backups: in /usr/local/lino/lino_local/prod/ -rw-rw---- 1 root www-data 8630781 May 25 06:29 snapshot.zip and in /var/backups/lino/prod/: -rw-rw---- 1 root www-data 8622994 May 24 06:30 20...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Yes, I also guess that it has to do with permissions. Our real problem is that we don't have any log message. It's kind of absurd that a script doesn't do its job and doesn't explain why.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : One interesting detail about linobackups in jane: if linobackups has rights -rwxr-xr-x, then each morning jane backup is done in /var/backups/lino/jane and /usr/local/lino/lino_local/jane. However, if linobackups has rights -rwx------, then backup in /var/backups/lino/jane wouldn't be done, but in ...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Why does it work on avanti but not on jane? On avanti we have a script /etc/cron.daily/make_snapshot_prod.sh, which calls /usr/local/lino/lino_local/prod/make_snapshot.sh. The current solution on jane is not okay because we do not want a third script.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Yes, now it works again. The question remains: why!?
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Restored working configuration for jane backups. Last backup for jane in /var/backups/lino/jane was at May 15. Next should be for May 23.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) (1 reply) : Log shows: May 21 05:25:01 laudate CRON[3451649]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) that cron.daily works at morning, also backup from jane is done and sent to bombur, but these 2 new scripts make_snapshot_ets.sh and make_snapshot_jane.sh don'...
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) (1 reply) : Changed these files so that only root has rights for the files (because crond runsas root). From unix.stackexchange.com/questions/272860/what-are-the-filesystem-permissions-for-a-cron-job "You can set the permissions on those files to deny read access to everyone, if you wish; it doesn't matter to ro...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : The question is: why?
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : As we see, jane stopped again making daily snapshots. The last snapshot is from May, 15 at 5.26.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : I also started a [url luc.lino-framework.org/blog/2024/0516.html#a-miracle-in-cron blog entry about this].
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Let's see whether the problem comes back.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : So I moved the two scripts back from /usr/local/bin to /etc/cron.daily and moved linobackups to ~/20240516:
$ cat linobackups #!/bin/sh /usr/local/bin/make_snapshot_ets.sh > /dev/null sleep 5 /usr/local/bin/make_snapshot_jane.sh > /dev/null $ mkdir ~/20240516 $ cd ~/20240516 $ sudo mv /etc/cron.dai...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Yes, the daily snapshots on laudate are agin being done. But I don't believe in miracles.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Looking in jane to /var/backups/lino/jane I see that after my last update, somehow the cron-backup started to work: we have backups done at 5.26 (e.g. time for cron.daily) for May 10, 11, 12 and 13. Let's see further results.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Hannes, look at the /etc/cron.d/pull_demos file on LF. It calls /usr/local/bin/pull_demos.sh (without redirecting anything) and this pull_demos.sh script then uses what we discovered recently. It redirects both stdout and stderr to to a same file, which it cats to stdout when the process exits with ...
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : There is still some issue with cron. The linobackups-script in /etc/cron.daily didn't create a backup this morning. I executed the script from commandline myself and then backups in /var/backups were created. So, the issue needs further study.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : In cron.daily created a script linobackups which runs make_snapshot_ets.sh and make_snapshot_jane.sh in /usr/local/bin. For testing purposes moved linobackups to cron.hourly and saw that 2 backups in /var/backups/lino were created. Then moved linobackups back to /etc/cron.daily and restarted cron.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : It's mysterious: the make_snapshot_jane.sh script in cron.daily runs normally when invoked manually, and /var/log/cron.log says that cron.daily been started today at 7:30 am, but the snapshot.zip had not been written. And no error message anywhere.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) (1 reply) : cat /etc/crontab tells us when and by who cron.daily should run: 25 5 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
But in laudate: ls /usr/sbin/anacron ls: cannot access '/usr/sbin/anacron': No such file or directory
Running separately from commandline...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : I activted cron logging by uncommenting the cron line in /etc/rsyslog.conf (as explained [url stackoverflow.com/questions/4883069/debugging-crontab-jobs here]).
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : The daily cron job ran this morning at 5:25, but snapshot.zip in jane is still from yesterday 21:42.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Did chown -c root:root for make_snapshot_ets.sh and make_snapshot_jane.sh, also restarted cron. Before that the ownerships were root:www-data for both. Lets see the result again. Executed these scripts separately from commandline and corresponding backups were created: in /usr/local/lino/lino_lo...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Didn't work: the snapshot.zip in /usr/local/lino/lino_local/jane is still the one I made yesterday manually using ./make_snapshot.py from the command-line.
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Maybe the fact that make_snapshot_ets.sh and make_snapshot_jane.sh were group-writable. Did chmod g-w for both and restarted cron. Lets see if that helps. If this is the case, group-writability should be removed from version update scripts.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Yes, the problem is not the borg backup to bombur but the daily cron job that calls make_snapshot. The job doesn't work any more. Very probably because of an upgrade i made that day. But what is the error message? And why didn't we get notified?
2 aastat, 5 kuud ago by hannes about #5588 (Jane is not making daily snapshots) : Mounted (in jane) backups with borg as: borg mount jane@bombur.saffre-rumma.net:/home/jane/backups /mnt/ Inside /mnt-directory ls gives laudate-2024-05-05T05:25:02 as the last backup. In bombur, /home/jane/backups I also see that last backup was done today morning. Of course, /var/backups/lino/j...
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) (1 reply) : We have been working without backup for almost two weeks.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : The last snapshot of Jane was 20240426_1844. Hannes, please have a look at why the daily snapshot doesn't work.
2 aastat, 5 kuud ago by Luc Saffre about #5588 (Jane is not making daily snapshots) : Created new Ticket #5588 (Jane is not making daily snapshots).