by Luc Saffre · in Developers · assigned to hannes

Summary

Children

• #5301 (kuurunner is offline) (by Luc Saffre in Hosting (system administration))

Comments

• 2 years, 8 months ago by hannes about #5295 (Is kuurunner down?) : Now kuurunner seems to be up and running, however pipelines run slowly.
• 2 years, 8 months ago by Luc Saffre about #5295 (Is kuurunner down?) : The runner was again not working, already two weeks. I rebooted kuu and restarted the last pipeline for lino, but it is still "in pending state and is waiting to be picked by a runner".
• 2 years, 10 months ago by hannes about #5295 (Is kuurunner down?) : Finally: dets 23 12:18:53 kuu gitlab-runner[532]: Removed job from processing list
• 2 years, 10 months ago by hannes about #5295 (Is kuurunner down?) : Restarted kuu, seems now gitlab processes spinning up again.
• 2 years, 10 months ago by hannes about #5295 (Is kuurunner down?) : There is only 1,8G  diskspace available in kuu (or 91% used). Could this be the reason why some big tasks will get stuck?
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : I added a few notes to [url hosting.lino-framework.org/gitlab_runner/#cheat-sheet our gitlab-runner cheat sheet].
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : After cleaning up it did not restart working. "gitlab-runner status" didn't report any problem. I restarted the gitlab-runner service and now it works again. And I still have no clue why...
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : I ran "docker system prune", which reclaimed 3.658GB of disk space (from a total of 20GB). Use% reported by df went from 92% to 73%
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : @8lurry suggested to clean up the dangling docker containers if there are too many and to check the available storage.
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : TODO: write more docs about monitoring our gitabl-runner
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : GitLab reported that it had no contact since 2 days. Sharif and I restarted the service on kuu, and now it seems to work again. I started a new section "Cheat sheet" in gitlab-runner.rst in the hg.
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : I will restart the gitlab-runner service on kuu (although it says active (running) since Sat 2023-10-21 12:21:42 CEST; 1 months 14 days ago), but there is something suspicious:
  luc@kuu:~$ sudo service gitlab-runner status ... dets 03 04:26:16 kuu gitlab-runner[277910]: Appending trace to coordinator...
• 2 years, 10 months ago by Luc Saffre about #5295 (Is kuurunner down?) : Created new Ticket #5295 (Is bombur-runner down?).