Repair-MongoDB.md 4.8 KB

Repair Snap WeKan

sudo su

snap stop wekan

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/snap/wekan/current/lib/x86_64-linux-gnu

export PATH="/snap/wekan/current/bin:$PATH"

mongod --dbpath "/var/snap/wekan/common" --repair >> /root/repairlog.txt

snap start wekan

exit

Logs from Snap:

https://github.com/wekan/wekan/issues/5073

Repair Docker WeKan

At step 12 below is actual repair MongoDB command.

Some customer of WeKan Commercial Support ordered restore and repair of WeKan MongoDB database at UCS appliance. This was needed when changing back to previous UCS 4.x major version, when in new major UCS 5.x version Docker containers were all the time crashing and restarting, or something else got broken. Here are commands that were used while repairing.

Similarly, MongoDB may require repair, if:

  • MongoDB does not recover from sudden power failure cleanly
  • Disk comes full, that is worst case. Recommended are daily backups, in this case sometimes repair does not work well enough.
  1. SSH to UCS appliance as root:

    ssh root@192.168.0.100
    
  2. If backup or raw database files is at external USB harddrive, look where it is inserted:

    ls /dev
    
    fdisk -l /dev/sdb
    
    fdisk -l /dev/sdc
    
    fdisk -l /dev/sdd
    
  3. If external USB drive has NTFS file format, and mount complains about not having NTFS support, install NTFS support to Linux:

    apt install ntfs-3g
    
  4. Mount USB drive to /mnt

    mount /dev/sdd1 /mnt
    
  5. Stop WeKan Docker containers. If copying raw database files, WeKan and MongoDB should not be running, so that MongoDB would not be even more corrupted. (If MongoDB is running, and you have mongodump backup, you can Backup with mongorestore

    docker stop wekan-db wekan-app
    
  6. Find WeKan database raw files. MongoDB has various database engines, like WiredTiger, how to save raw data compressed, or other formats. WiredTiger is file format to save compressed MongoDB data. Similar like MySQL has ISAM, InnoDB etc.

    apt -y install mlocate
    updatedb
    locate WiredTiger.wt
    

    It can show for example USB drive and Docker container directory:

    root@ucs:/mnt/wekan/data/db# locate WiredTiger.wt
    /mnt/wekan/data/db/WiredTiger.wt
    /var/lib/univention-appcenter/apps/wekan/data/db/WiredTiger.wt
    
  7. Change to newly installed MongoDB data directory:

    cd /var/lib/univention-appcenter/apps/wekan/data/db/
    
  8. Look at directory files owner permissions:

    ls -lah
    

    There could be like this:

    -rw-r--r-- 1 tss  tss  1004 Jan 25 13:09 WiredTiger.turtle
    -rw-r--r-- 1 tss  tss  392K Jan 25 13:09 WiredTiger.wt
    
  9. Move that probably empty newly installed MongoDB data elsewhere:

    mkdir /root/new-empty-wekan-data
    mv * /root/new-empty-wekan-data
    
  10. Copy MongoDB raw database files from USB harddrive to docker container directory:

    root@ucs-bdc:/var/lib/univention-appcenter/apps/wekan/data/db# cp -pR /mnt/wekan/data/db/* .
    
  11. Like looked at step 7, change file owner permissions to be correct, and change directory

    chown -R tss:tss *
    cd /root
    
  12. Find mongod command:

    locate /usr/bin/mongod
    
  13. Repair Docker WeKan with version of Docker MongoDB that WeKan uses, change mongod path to below. Repairing logged to textfile.

    root@ucs:~# /var/lib/docker/overlay2/7b58483a16a2f67ee50486c00ec669940f7a95d460ee8188966fee0096e81fa2/diff/usr/bin/mongod --dbpath "/var/lib/univention-appcenter/apps/wekan/data/db" --repair >> repairlog.txt
    
  14. Look what containers are running:

    docker ps
    
  15. Look what containers are not running:

    docker ps -a
    
  16. Start MongoDB first:

    docker start wekan-db
    
  17. Look what happends at MongoDB container:

    docker logs wekan-db
    
  18. Start WeKan next.

    docker start wekan-app
    
  19. Look what happends at MongoDB container.

    docker logs wekan-app
    
  20. Check are both wekan-db and wekan-app containers running, and not restarting:

    docker ps
    
  21. Try to login to WeKan with webbrowser.

  22. Backup WeKan database now after repair:

    docker exec -it wekan-db bash
    cd /data
    rm -rf dump
    mongodump
    exit
    docker cp wekan-db:/data/dump .
    zip -r wekan-backup-YEAR-MONTH-DATE-TIME-HERE.zip dump
    

    Now backup is at wekan-backup-YEAR-MONTH-DATE-TIME-HERE.zip file.

  23. At your local computer terminal (not at UCS server), transfer file to your local computer with scp:

    scp root@192.168.0.100:/root/wekan-backup-YEAR-MONTH-DATE-TIME-HERE.zip .
    
  24. Copy backup to external USB harddrive, change YEAR-MONTH-DATE-TIME-HERE to current date and time:

    cp /root/wekan-backup-YEAR-MONTH-DATE-TIME-HERE.zip /mnt/
    
  25. Look what drives are mounted:

    df -h
    
  26. Sync and save unwritted data to disk, and unmount external USB harddrive safely:

    sync
    umount /dev/sdd1
    
  27. Remove external USB harddisk from server.

  28. Login to WeKan, check do all WeKan boards work. In this case, all did work.