{"id":2422,"date":"2015-10-06T20:21:25","date_gmt":"2015-10-06T19:21:25","guid":{"rendered":"http:\/\/roboblog.fatal-fury.de\/?p=2422"},"modified":"2025-04-15T11:02:44","modified_gmt":"2025-04-15T10:02:44","slug":"infiniband-cluster","status":"publish","type":"post","link":"http:\/\/roboblog.fatal-fury.de\/?p=2422","title":{"rendered":"Infiniband Cluster"},"content":{"rendered":"<p><a href=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster5.jpg\" rel=\"attachment wp-att-4425\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster5-150x150.jpg\" alt=\"IBcluster5\" width=\"150\" height=\"150\" class=\"alignnone size-thumbnail wp-image-4425\" \/><\/a><\/p>\n<p><a href=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster2.jpg\" rel=\"attachment wp-att-4179\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster2-150x150.jpg\" alt=\"IBcluster2\" width=\"150\" height=\"150\" class=\"alignnone size-thumbnail wp-image-4179\" \/><\/a><\/p>\n<p><a href=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/IBcluster-150x150.jpg\" alt=\"IBcluster\" width=\"150\" height=\"150\" class=\"alignnone size-thumbnail wp-image-2423\" \/><\/a><\/p>\n<p><a href=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/staubfilter1.jpg\" rel=\"attachment wp-att-4559\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/roboblog.fatal-fury.de\/wp-content\/uploads\/2015\/10\/staubfilter1-150x150.jpg\" alt=\"staubfilter1\" width=\"150\" height=\"150\" class=\"alignnone size-thumbnail wp-image-4559\" \/><\/a><\/p>\n<p>LD_LIBRARY_PATH=\/home\/kater\/bin\/ las2txt64  -parse xyz -odir . -i 47605560_all_thin.las -keep_class  2<\/p>\n<h1>GIT<\/h1>\n<p>Repair Permissions<br \/>\n  The repository isn't configured to be a shared repository (see core.sharedRepository in git help config). If the output of:<br \/>\n  git config core.sharedRepository<br \/>\n  is not group or true or 1 or some mask, try running:<br \/>\n  git config core.sharedRepository group<\/p>\n<pre>\r\ncd \/path\/to\/repo.git\r\nchgrp -R groupname .\r\nchmod -R g+rwX .\r\nfind . -type d -exec chmod g+s '{}' +\r\n<\/pre>\n<p>GIT Branch dauerhaft anzeigen.  in .bash_rc<\/p>\n<p>parse_git_branch() {<br \/>\n     git branch 2> \/dev\/null | sed -e '\/^[^*]\/d' -e 's\/* <span class='MathJax_Preview'><img src='http:\/\/roboblog.fatal-fury.de\/wp-content\/plugins\/latex\/cache\/tex_4fd4a56c435d5c26b3c3c366f51b074e.gif' style='vertical-align: middle; border: none; ' class='tex' alt=\"\" \/><\/span><script type='math\/tex'><\/script>\/(\\1)\/'<br \/>\n}<br \/>\nexport PS1=\"\\u@\\h <p style='text-align:center;'><span class='MathJax_Preview'><img src='http:\/\/roboblog.fatal-fury.de\/wp-content\/plugins\/latex\/cache\/tex_3ad3c9d195d2c4d566e10816edd6e80d.gif' style='vertical-align: middle; border: none;' class='tex' alt=\"\" \/><\/span><script type='math\/tex;  mode=display'><\/script><\/p>\\w <p style='text-align:center;'><span class='MathJax_Preview'><img src='http:\/\/roboblog.fatal-fury.de\/wp-content\/plugins\/latex\/cache\/tex_d577b32aa03ba4479a540c21b9387d82.gif' style='vertical-align: middle; border: none;' class='tex' alt=\"\" \/><\/span><script type='math\/tex;  mode=display'><\/script><\/p>\\$(parse_git_branch)<p style='text-align:center;'><span class='MathJax_Preview'><img src='http:\/\/roboblog.fatal-fury.de\/wp-content\/plugins\/latex\/cache\/tex_59f95c218ac799e4ec70953fd8d7ae72.gif' style='vertical-align: middle; border: none;' class='tex' alt=\"\" \/><\/span><script type='math\/tex;  mode=display'><\/script><\/p>$ \"<\/p>\n<h1>samba SMB<\/h1>\n<p>To enable old Version 1 SMB protocol version add vers=1.0<br \/>\nDie user Option hat eine doppelte Bedeutung. Steht sie allein, werden keine root Rechte zum mounten ben\u00f6tigt. Wird ein username bei user= angegeben, so bezieht sich dieser auf den Server. Die Option users mit einem s am Ende erlaubt, dass keine root Rechte ben\u00f6tigt werden und dass alle user die Partition umounten kann.<br \/>\nuid=<br \/>\ngid=<\/p>\n<p>F\u00fcr Multiusersysteme ist die Option noperm f\u00fcr cifs mounts hilfreich.<br \/>\nSchreibrechte checken!<\/p>\n<pre>\r\n192.168.10.49:\/perm     \/perm           nfs     noauto,nosuid,hard,user         0       0\r\n192.168.10.49:\/karten   \/karten         nfs     noauto,nosuid,hard,user         0       0\r\n\r\n\/\/192.168.10.13\/data             \/laufwerkG      cifs    noauto,hard,users,user=standard,password=standard,vers=1.0,uid=1000,gid=998,noperm,exec       0       0\r\n\r\n\r\n<\/pre>\n<h1>IGM samba SMB<\/h1>\n<p>Neuen User mit adduser anlegen. Default group auf users \u00e4ndern<br \/>\n# sudo usermod -g users mhu<br \/>\nIn Webmin neuen User convertieren<br \/>\nIn Webmin Service restart<br \/>\nPoint 139 445 forwarden<br \/>\nIn Konquerror smb:\/\/mhu@localhost\/<br \/>\noder in der Konsole sudo mount -t cifs -o user=mhu \/\/localhost\/IGM-2 mnt\/<br \/>\ntesten<\/p>\n<h1>RAID<\/h1>\n<p>Allgemeiner Status:<br \/>\n$ cat \/proc\/mdstat<br \/>\nDie Angaben in dne [] Klammern sind Informationen pro Platte. Z.B. [UU] f\u00fcr 2 Platten.<br \/>\nU - Up<br \/>\nS - spare (schonen)<br \/>\n_ - Down<\/p>\n<p>Informationen zum einzelnen Raid<br \/>\n$ mdadm -D \/dev\/md0<\/p>\n<p>Hm immer gut<br \/>\n$mdadm --assemble --scan -v<\/p>\n<p>Jede Partition einzeln untersuchen<br \/>\n$ mdadm -E \/dev\/sda1<br \/>\n$ mdadm -E \/dev\/sdb1<\/p>\n<p>RAID Einrichten:<br \/>\nhttps:\/\/www.thomas-krenn.com\/de\/wiki\/Linux_Software_RAID<\/p>\n<p>Partition auf jede Festplatte erstellen mit Type Linux Raid<br \/>\nRAID mit mdadm erstellen<br \/>\nmdadm --create \/dev\/md0 --level=0 --raid-devices=3 \/dev\/sdb1 \/dev\/sdc1 \/dev\/sdd1<\/p>\n<p>Dateisystem erstellen<br \/>\nmkfs.ext4 \/dev\/md0<\/p>\n<p>UUID einer Festplatte herausfinden<br \/>\nroot@login02:~#  blkid \/dev\/sda<br \/>\n\/dev\/sda: UUID=\"7578c0b0-8fbe-44ba-93f4-cd55fe64e220\" BLOCK_SIZE=\"4096\" TYPE=\"ext4\"<\/p>\n<h1>BACKUP and EMAIL notify<\/h1>\n<p>Backup Script:<br \/>\nrsync --numeric-ids  --stats -hPave ssh \/IGM2 192.168.:\/\/share\/CACHEDEV1_DATA\/data  > `date \"+%y%m%d\"_copyprotocol`<br \/>\necho -e \"Subject: Backup PS2NAS\\n\\n\" > tmp; cat `date \"+%y%m%d\"_copyprotocol` >> tmp; cat tmp | msmtp mail@me<\/p>\n<p>Bei Fehler:<br \/>\nmsmtp: TLS certificate verification failed: the certificate fingerprint does not match<br \/>\nmsmtp: could not send mail (account default from \/root\/.msmtprc)<\/p>\n<p>Das tun:<br \/>\nmsmtp --serverinfo --tls --tls-certcheck=off --host=smtp.1und1.de --port 587<\/p>\n<p>Und du bekommst:<br \/>\nSHA256: 6F:5C:80:76:FB:8A:82:02:E9:23:AB:4A:68:F4:E2:6C:B6:29:72:70:82:5D:02:F8:F5:A5:AB:AC:67:D3:72:61<\/p>\n<h1>Logical Volume Management<\/h1>\n<p>$ vgdisplay<br \/>\n  Alloc PE \/ Size       29440 \/ 115.00 GiB<br \/>\n  Free  PE \/ Size       207491 \/ 810.51 GiB<\/p>\n<p>$ lvextend -L910.51G \/dev\/mapper\/vg00-home<br \/>\n  Rounding size to boundary between physical extents: 910.51 GiB<br \/>\n  Size of logical volume vg00\/home changed from 100.00 GiB (25600 extents) to 910.51 GiB (233091 extents).<br \/>\n  Logical volume home successfully resized.<\/p>\n<p>$ resize2fs \/dev\/mapper\/vg00-home<br \/>\nFilesystem at \/dev\/mapper\/vg00-home is mounted on \/home; on-line resizing required<br \/>\nold_desc_blocks = 7, new_desc_blocks = 57<br \/>\nThe filesystem on \/dev\/mapper\/vg00-home is now 238685184 (4k) blocks long.<\/p>\n<p>$ df -h<br \/>\n\/dev\/mapper\/vg00-home  897G  560M  860G   1% \/home<\/p>\n<h1>Software<\/h1>\n<p>Put<br \/>\napt::install-recommends \"false\";<br \/>\nin \/etc\/apt\/apt.conf after installed basesystem to get rid of stupid nonsense software on a server.<br \/>\nCheck with \"aptitude why \" and  apt-config dump |fgrep -i recommend<\/p>\n<hr>\n<p>Don't install these<\/p>\n<blockquote><p>kde-runtime\n<\/p><\/blockquote>\n<p>Don't deinstall<\/p>\n<blockquote><p>  # IB<br \/>\nlibmlx4-1 libmthca1<\/p>\n<p># obviously<br \/>\nmeld  <\/p>\n<p># for deptherror script<br \/>\npython-matplotlib<br \/>\npython-numpy<br \/>\npython-tk\n<\/p><\/blockquote>\n<p>Check for<\/p>\n<blockquote><p>\n*kde*<br \/>\n*gnome*<br \/>\n*gtk*<br \/>\n*vlc*<br \/>\n*wx*<br \/>\nlibqt4*-dev<br \/>\nlibqt4* keep for all quads<br \/>\nlibqt5*\n<\/p><\/blockquote>\n<hr>\n<p># checkinstall : um aus dem Quelltext ein Debian-Paket (.deb) zu erstellen,<\/p>\n<p>Desktop Rechner<br \/>\napt-get install konsole kate libreoffice kcalc konqueror cifs-utils nfs-common lm-sensors  im net-tools <\/p>\n<p>Cluster<br \/>\napt-get install  flex  build-essential libhwloc5 bclibnuma1 ganglia-monitor  apt-file ipmitool libconfuse-dev<\/p>\n<pre>\r\n\/etc\/modules\r\nipmi_devintf\r\n<\/pre>\n<hr>\n<p># timeserver<br \/>\napt-get install ntpdate<br \/>\n$ cat \/etc\/cron.daily\/ntp<br \/>\n#!\/bin\/sh<br \/>\nntpdate time.fu-berlin.de<\/p>\n<p>chmod +x \/etc\/cron.daily\/ntp<\/p>\n<hr>\n<p># stuff you prob. want to deinstall<br \/>\nmlocale collectd*<\/p>\n<h1>Cluster Allgemein<\/h1>\n<p>Ergebnise Kopieren (need update)<br \/>\n$ rsync -aPz ergzus.bin ergqh.bin ergqhmax.bin *.nml *.txt fort.4242 abenheim3979.* hydstructs thomas@.ddns.net:\/hastenichtgesehn\/<\/p>\n<p><textarea readonly># \/usr\/share\/polkit-1\/actions\/org.freedesktop.login1.policy<br \/>\norg.freedesktop.login1.set-user-linger\"<br \/>\n                        <allow_any>yes<\/allow_any><br \/>\n                        <allow_inactive>yes<\/allow_inactive><br \/>\n                        <allow_active>yes<\/allow_active><\/textarea><\/p>\n<h1>MPI Debuggen<\/h1>\n<p>Startet fuer jeden Thread ein xterm Fenster.<br \/>\n--ex run Startet den Debugger<\/p>\n<pre><code>\r\nmpiexec  -n 16 xterm -e gdb -ex \"run\" --args  \/home\/kater\/bin\/unrunoff\r\n<\/code><\/pre>\n<h1>Infiniband<\/h1>\n<p>F\u00fcr die neuen Karten in quad21-26<br \/>\n81:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB]<\/p>\n<p>Firmeware update der Karte machen, Siehe https:\/\/www.thomas-krenn.com\/de\/wikiDE\/index.php?title=Mellanox_Firmware_Tools_-_Firmware_Upgrade_unter_Linux&xtxsearchselecthit=1<\/p>\n<p>mft-4.23.0-104-x86_64-deb.tgz runterladen, entpacken, installieren. Daf\u00fcr folgende Pakete installieren:<br \/>\napt-get install gcc make dkms linux-headers-5.10.0-19-amd64 libucx0 libgfortran5  libevent-pthreads-2.1-7 libevent-core-2.1-7 libhwloc15 htop<\/p>\n<p>Mit mlxfwmanager Firmware Version feststellen<br \/>\nNeue Firmaware runterladen:<br \/>\nmlxfwmanager --download-os linux_x64 --download-type self_extractor -y<br \/>\ncd linux_x64\/<br \/>\nlinux_x64# .\/mlxup<br \/>\n  Versions:         Current        Available<br \/>\n     FW             10.10.4020     10.16.1200<br \/>\nreboot<br \/>\nroot@quad21:~# ibstatus<br \/>\nInfiniband device 'mlx5_0' port 1 status:<br \/>\n        default gid:     fe80:0000:0000:0000:e41d:2d03:000c:0c20<br \/>\n        base lid:        0x4<br \/>\n        sm lid:          0x1<br \/>\n        state:           4: ACTIVE<br \/>\n        phys state:      5: LinkUp<br \/>\n        rate:            40 Gb\/sec (4X FDR10)<br \/>\n        link_layer:      InfiniBand<\/p>\n<p>====================================<\/p>\n<p>https:\/\/wiki.debian.org\/RDMA<br \/>\nInstall rdma-core<\/p>\n<p>  Mellanox Technologies MT26428 Installation<\/p>\n<p>apt-get install ibutils infiniband-diags perftest  libmlx4-1 libmthca1 libipathverbs1  rdmacm-utils  cpufrequtils ibverbs-utils  infiniband-diags perftest  firmware-linux-free<br \/>\n### neueres? \u00e4lters? paket: ibverbs-providers<\/p>\n<p>modprobe ib_umad ib_mthca mlx4_ib ib_uverbs ib_ipoib rdma_ucm  mlx4_core  (Module einzeln laden)<\/p>\n<pre>\r\n\/etc\/modules\r\nib_umad\r\nib_mthca\r\nmlx4_ib\r\nib_uverbs\r\nib_ipoib\r\nrdma_ucm \r\nmlx4_core\r\n<\/pre>\n<hr>\n<p>(Ne brauchen wir nicht)<br \/>\nUses who want to run MPI jobs will need to have write permissions for the following devices:<br \/>\nThe simplest way to do this is to add the users to the rdma group.<\/p>\n<pre>\r\n\/dev\/infiniband\/uverbs*\r\n\/dev\/infiniband\/rdma_cm*\r\nusermod -a -G rdma aron\r\n<\/pre>\n<hr>\n<p>OpenMPI will need to pin memory. Edit \/etc\/security\/limits.conf and add the line:<\/p>\n<pre>\r\n* hard memlock unlimited \r\n* soft memlock unlimited \r\n<\/pre>\n<hr>\n<p># to ping start server and use port GUID from ibstat<br \/>\nibping -S<br \/>\nibping -G fe80::2:c903:10:5fa4<br \/>\nPong from quad02.(none) (Lid 5): time 0.602 ms<\/p>\n<hr>\n<p>Programme:<br \/>\n$ ibdiagnet<br \/>\n$ iblinkinfo<br \/>\n$ ibnetdiscover  - to get lid<br \/>\n$ ibportstate to set links speed<br \/>\n lid port speed speed<br \/>\nspeed is the speed of the port: 1 for 2.5 Gbyte\/sec, 2 for 5.0 Gbyte\/sec, and 4 for 10.0 Gbyte\/sec.<br \/>\nibportstate  14 1 speed 7<br \/>\noperation \"enable\" hilft um den link speed zu aktivieren<br \/>\n$ perfquery<\/p>\n<hr>\n<h1>Ganglia<\/h1>\n<p>IB plugin https:\/\/github.com\/ULHPC\/ganglia_infiniband_module<br \/>\nDamit ganglia perfquery ausf\u00fchren kann ohne sudo<br \/>\n<code>$ chmod u+s \/usr\/sbin\/perfquery<\/code><\/p>\n<p>Braucht Symlink von libconfuse.so.0<br \/>\nthomas@quad11:\/usr\/lib\/x86_64-linux-gnu$ ll libconfuse.*<br \/>\n-rw-r--r-- 1 root root 51K Aug 15  2018 libconfuse.so.1.0.0<br \/>\nlrwxrwxrwx 1 root root  19 Aug 15  2018 libconfuse.so -> libconfuse.so.1.0.0<br \/>\nlrwxrwxrwx 1 root root  13 Jul 24 09:11 libconfuse.so.0 -> libconfuse.so<\/p>\n<p>\/etc\/ganglia\/gmond.conf <\/p>\n<pre>\r\nmodules {\r\n   module {\r\n     name = \"ib_module\"\r\n     language = \"C\/C++\"\r\n     path = \"\/root\/modInfiniband.so\"\r\n  }\r\n}\r\n\r\ncollection_group {\r\n  collect_every = 20\r\n  time_threshold = 60\r\n  metric {\r\n    name = \"ib_bytes_out\"\r\n    value_threshold = 1000000000000\r\n    title = \"Bytes Sent(infiniband)\"\r\n  }\r\n  metric {\r\n    name = \"ib_bytes_in\"\r\n    value_threshold = 1000000000000\r\n    title = \"Bytes Received(infiniband)\"\r\n  }\r\n  metric {\r\n    name = \"ib_pkts_in\"\r\n    value_threshold = 100000000000\r\n    title = \"Packets Received(infiniband)\"\r\n  }\r\n  metric {\r\n    name = \"ib_pkts_out\"\r\n    value_threshold = 100000000000\r\n    title = \"Packets Sent(infiniband)\"\r\n  }\r\n}\r\n\r\n<\/pre>\n<p>crontab -e<br \/>\n* * * * * \/home\/thomas\/bin\/gmetric-cpu-temp.sh <\/p>\n<p>Trouble<br \/>\n=======<\/p>\n<p>[quad01:05373] Error: unknown option \"--hnp-topo-sig\"<br \/>\nUse full path for mpiexec and inka_kopplung<\/p>\n<pre>\r\nlatency and bandwidth test\r\n==========================\r\n# make sure every CPU has the same FREQ because stupidity\r\nfor ((i=0; i<32;i++)); do cpufreq-set -g performance -c $i; done\r\n# On one PC\r\nib_send_lat -a\r\n# On other PC\r\nib_send_lat 192.168.20.112 -a\r\n# Same for ib_send_bw\r\n<\/pre>\n<pre>\r\nib_send_lat -a 192.168.20.112 \r\n---------------------------------------------------------------------------------------\r\n                    Send Latency Test\r\n Dual-port       : OFF          Device         : mlx4_0\r\n Number of qps   : 1            Transport type : IB\r\n Connection type : RC           Using SRQ      : OFF\r\n TX depth        : 1\r\n Mtu             : 4096[B]\r\n Link type       : IB\r\n Max inline data : 236[B]\r\n rdma_cm QPs     : OFF\r\n Data ex. method : Ethernet\r\n---------------------------------------------------------------------------------------\r\n local address: LID 0x07 QPN 0x0145 PSN 0x40bbdc\r\n remote address: LID 0x08 QPN 0x0127 PSN 0x625da0\r\n---------------------------------------------------------------------------------------\r\n #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]\r\n 2       1000          1.32           29.75        1.98   \r\n 4       1000          1.24           15.99        1.54   \r\n 8       1000          1.24           8.66         1.42   \r\n 16      1000          1.19           11.14        1.40   \r\n 32      1000          1.24           12.68        1.58   \r\n 64      1000          1.37           13.84        1.66   \r\n 128     1000          1.53           14.81        1.83   \r\n 256     1000          3.08           33.22        5.29   \r\n 512     1000          3.25           19.88        4.01   \r\n 1024    1000          3.73           14.94        4.12   \r\n 2048    1000          5.21           14.80        5.57   \r\n 4096    1000          8.23           22.35        8.64   \r\n 8192    1000          10.94          32.77        11.47  \r\n 16384   1000          16.45          49.51        17.58  \r\n 32768   1000          27.41          74.35        29.15  \r\n 65536   1000          50.14          147.59       51.21  \r\n 131072  1000          94.49          268.19       96.09  \r\n 262144  1000          183.19         472.53       184.69 \r\n 524288  1000          360.21         832.79       365.68 \r\n 1048576 1000          714.91         1399.75      721.41 \r\n 2097152 1000          1422.95        2865.38      1437.44\r\n 4194304 1000          2840.72        4905.65      2907.37\r\n 8388608 1000          5807.65        10473.58      5936.33\r\n---------------------------------------------------------------------------------------\r\n\r\nib_send_bw 192.168.20.112  -a\r\n---------------------------------------------------------------------------------------\r\n                    Send BW Test\r\n Dual-port       : OFF          Device         : mlx4_0\r\n Number of qps   : 1            Transport type : IB\r\n Connection type : RC           Using SRQ      : OFF\r\n TX depth        : 128\r\n CQ Moderation   : 100\r\n Mtu             : 4096[B]\r\n Link type       : IB\r\n Max inline data : 0[B]\r\n rdma_cm QPs     : OFF\r\n Data ex. method : Ethernet\r\n---------------------------------------------------------------------------------------\r\n local address: LID 0x07 QPN 0x0148 PSN 0xd4c92e\r\n remote address: LID 0x08 QPN 0x012a PSN 0x51f56e\r\n---------------------------------------------------------------------------------------\r\n #bytes     #iterations    BW peak[MB\/sec]    BW average[MB\/sec]   MsgRate[Mpps]\r\n 2          1000             7.64               5.88               3.080942\r\n 4          1000             7.48               6.75               1.769229\r\n 8          1000             48.74              42.44              5.562193\r\n 16         1000             42.13              39.38              2.580683\r\n 32         1000             170.37             140.12             4.591350\r\n 64         1000             248.02             185.61             3.041025\r\n 128        1000             92.72              55.54              0.454995\r\n 256        1000             500.02             435.57             1.784099\r\n 512        1000             1109.73            1050.87            2.152184\r\n 1024       1000             1237.52            1217.68            1.246901\r\n 2048       1000             1267.55            1251.44            0.640736\r\n 4096       1000             1342.55            1336.95            0.342258\r\n 8192       1000             1371.66            1354.30            0.173351\r\n 16384      1000             1299.78            1183.89            0.075769\r\n 32768      1000             1311.13            1302.48            0.041679\r\n 65536      1000             1361.31            1361.30            0.021781\r\n 131072     1000             1387.05            1281.65            0.010253\r\n 262144     1000             1354.87            1291.06            0.005164\r\n 524288     1000             1311.00            1297.36            0.002595\r\n 1048576    1000             1344.04            1312.48            0.001312\r\n 2097152    1000             1320.18            1312.31            0.000656\r\n 4194304    1000             1349.24            1328.86            0.000332\r\n 8388608    1000             1281.43            1281.43            0.000160\r\n---------------------------------------------------------------------------------------\r\n<\/pre>\n<h1>FTP<\/h1>\n<p>very secure ftp vsftp<\/p>\n<pre>\r\nchroot_local_user=YES\r\nseccomp_sandbox=no\r\nuserlist_deny=NO\r\nuserlist_enable=YES\r\nuserlist_file=\/etc\/vsftpd.user_list\r\ncheck_shell=NO\r\nlocal_root=\/srv\/ftp\/\r\n<\/pre>\n<h1>Config Datein<\/h1>\n<p># IPv6 deaktivieren<br \/>\n$ cat \/etc\/sysctl.d\/01-disable-ipv6.conf<br \/>\nnet.ipv6.conf.all.disable_ipv6 = 1<\/p>\n<p># sshd_config<br \/>\nPermitRootLogin  no<br \/>\nStrictModes no<br \/>\nPubkeyAuthentication yes<br \/>\nPermitEmptyPasswords no<br \/>\nX11Forwarding yes<\/p>\n<h1>User Stuff<\/h1>\n<p>Sicherstellen, dass sich der User von jeden Quad auf jeden Quad automatisch einloggen kann. Da orted sich Baum-artig von Host zu Host einlogt.<\/p>\n<hr>\n<p>adduser --uid 1001 thomas<br \/>\naddgroup --gid 998 quad<br \/>\n# Change default group of user<br \/>\nusermod -g quad aron<br \/>\nusermod -g quad thomas<\/p>\n<hr>\n<p>## ~\/.bashrc<br \/>\n# stupid systemd idiots<br \/>\n# Nein der Scheiss bewirkt absolut garnichts! Nur screen hilft.<br \/>\n#loginctl enable-linger $USER <\/p>\n<p>#Change default file permission to rwx for directorys and files in<br \/>\numask 0002<\/p>\n<hr>\n<p>screen skript das immer die schon offenen session startet<br \/>\n<code><br \/>\n#!\/bin\/sh<\/p>\n<p>if [ $# -eq 0 ];  then<br \/>\n  #echo \"No arguments supplied\"<\/p>\n<p>  if \/usr\/bin\/screen -x; then<br \/>\n    echo \"\"<br \/>\n  else<br \/>\n    \/usr\/bin\/screen<br \/>\n fi<br \/>\nelse<br \/>\n  exec \/usr\/bin\/screen \"$@\"<br \/>\nfi<\/p>\n<p><\/code><\/p>\n<hr>\n<p># net quad00<br \/>\necho 1 > \/proc\/sys\/net\/ipv4\/ip_forward<br \/>\niptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE <\/p>\n<p># mpich debug<br \/>\n.\/configure --prefix=\/home6\/datahome\/aroland\/opt\/mpich-3.2.1_gcc-7.2.0_dbg\/ --enable-fortran=all --enable-g=all --enable-fast=O0 --enable-error-checking=all --enable-error-messages=all --enable-debuginfo<\/p>\n<p># netcdf hdf5<br \/>\nFC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpif90 CC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpicc .\/configure --prefix=\/home\/zanke\/opt\/hdf5-1.8.19_gfortran6.3.0\/ --enable-fortran --enable-fortran2003 --enable-parallel<br \/>\nmake<br \/>\nmake check<\/p>\n<p># netcdf<br \/>\nFC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpif90 CC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpicc CPPFLAGS=-I\/home\/zanke\/opt\/hdf5-1.8.19_gfortran6.3.0\/include LDFLAGS=-L\/home\/zanke\/opt\/hdf5-1.8.19_gfortran6.3.0\/lib   .\/configure --prefix=\/home\/zanke\/opt\/netcdf-4.4.1_gfortran6.3.0\/ <\/p>\n<p># netcdff<br \/>\nFC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpif90 CC=\/home\/zanke\/opt\/openmpi_gfortran\/bin\/mpicc CPPFLAGS=-I\/home\/zanke\/opt\/netcdf-4.4.1_gfortran6.3.0\/include LDFLAGS=-L\/home\/zanke\/opt\/netcdf-4.4.1_gfortran6.3.0\/lib   .\/configure --prefix=\/home\/zanke\/opt\/netcdf-4.4.1_gfortran6.3.0\/ <\/p>\n<h1>S.M.A.R.T<\/h1>\n<p>Wichtige Werte f\u00fcr nicht SSD Platten, die eine Langzeit \u00dcberwachung mit z.B. Ganglia brauchen IMO:<br \/>\n188 Command_Timeout<br \/>\n197 Current_Pending_Sector<br \/>\n198 Offline_Uncorrectable<br \/>\n5 Reallocated_Sector_Ct<br \/>\n187 Reported_Uncorrect<br \/>\n10 Spin_Retry_Count<br \/>\n199 UDMA_CRC_Error_Count<\/p>\n<p># smartctl -a \/dev\/sda | egrep  \"Command_Timeout|Current_Pending_Sector|Offline_Uncorrectable|Reallocated_Sector_Ct|Reported_Uncorrect|Spin_Retry_Count|UDMA_CRC_Error_Count\"<\/p>\n<p>29.10.2019<\/p>\n<p>sda \/ \/home<\/p>\n<pre>\r\n=== START OF INFORMATION SECTION ===\r\nDevice Model:     ST8000AS0002-1NA17Z\r\nSerial Number:    Z840AG52\r\nLU WWN Device Id: 5 000c50 08734ecdb\r\nFirmware Version: AR15\r\nUser Capacity:    8,001,563,222,016 bytes [8.00 TB]\r\nSector Sizes:     512 bytes logical, 4096 bytes physical\r\nRotation Rate:    5980 rpm\r\nDevice is:        Not in smartctl database [for details use: -P showall]\r\nATA Version is:   ACS-2, ACS-3 T13\/2161-D revision 3b\r\nSATA Version is:  SATA 3.1, 6.0 Gb\/s (current: 6.0 Gb\/s)\r\nLocal Time is:    Tue Oct 29 06:59:44 2019 CET\r\n\r\nSMART Attributes Data Structure revision number: 10\r\nVendor Specific SMART Attributes with Thresholds:\r\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\r\n  1 Raw_Read_Error_Rate     0x000f   114   099   006    Pre-fail  Always       -       72037664\r\n  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0\r\n  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       43\r\n  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0\r\n  7 Seek_Error_Rate         0x000f   084   060   030    Pre-fail  Always       -       50041550840\r\n  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       33792\r\n 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0\r\n 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       43\r\n183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0\r\n184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0\r\n187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0\r\n188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0\r\n189 High_Fly_Writes         0x003a   091   091   000    Old_age   Always       -       9\r\n190 Airflow_Temperature_Cel 0x0022   072   050   045    Old_age   Always       -       28 (Min\/Max 28\/36)\r\n191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0\r\n192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       1180\r\n193 Load_Cycle_Count        0x0032   095   095   000    Old_age   Always       -       11969\r\n194 Temperature_Celsius     0x0022   028   050   000    Old_age   Always       -       28 (0 17 0 0 0)\r\n195 Hardware_ECC_Recovered  0x001a   114   099   000    Old_age   Always       -       72037664\r\n197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0\r\n198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0\r\n199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0\r\n240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       31769 (216 109 0)\r\n241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       43960187200\r\n242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       35616462371\r\n\r\nSMART Error Log Version: 1\r\nNo Errors Logged\r\n\r\nSMART Self-test log structure revision number 1\r\nNum  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error\r\n# 1  Extended offline    Completed without error       00%     25167         -\r\n# 2  Extended offline    Completed without error       00%     22711         -\r\n<\/pre>\n<p>sdb \/perm<\/p>\n<pre>\r\n=== START OF INFORMATION SECTION ===\r\nDevice Model:     ST8000AS0002-1NA17Z\r\nSerial Number:    Z840AMJ5\r\nLU WWN Device Id: 5 000c50 08744d481\r\nFirmware Version: AR15\r\nUser Capacity:    8,001,563,222,016 bytes [8.00 TB]\r\nSector Sizes:     512 bytes logical, 4096 bytes physical\r\nRotation Rate:    5980 rpm\r\nDevice is:        Not in smartctl database [for details use: -P showall]\r\nATA Version is:   ACS-2, ACS-3 T13\/2161-D revision 3b\r\nSATA Version is:  SATA 3.1, 6.0 Gb\/s (current: 6.0 Gb\/s)\r\nLocal Time is:    Tue Oct 29 07:00:52 2019 CET\r\n\r\nSMART Attributes Data Structure revision number: 10\r\nVendor Specific SMART Attributes with Thresholds:\r\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\r\n  1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       187522464\r\n  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0\r\n  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       43\r\n  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0\r\n  7 Seek_Error_Rate         0x000f   087   060   030    Pre-fail  Always       -       4888765593\r\n  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       33791\r\n 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0\r\n 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       43\r\n183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0\r\n184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0\r\n187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0\r\n188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1\r\n189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0\r\n190 Airflow_Temperature_Cel 0x0022   071   049   045    Old_age   Always       -       29 (Min\/Max 28\/38)\r\n191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0\r\n192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       289\r\n193 Load_Cycle_Count        0x0032   088   088   000    Old_age   Always       -       25492\r\n194 Temperature_Celsius     0x0022   029   051   000    Old_age   Always       -       29 (0 18 0 0 0)\r\n195 Hardware_ECC_Recovered  0x001a   118   099   000    Old_age   Always       -       187522464\r\n197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0\r\n198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0\r\n199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0\r\n240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       22376 (153 118 0)\r\n241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       96619086856\r\n242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       63050316253\r\n<\/pre>\n<p>sdc \/karten<\/p>\n<pre>\r\n=== START OF INFORMATION SECTION ===\r\nDevice Model:     ST8000AS0002-1NA17Z\r\nSerial Number:    Z840AF12\r\nLU WWN Device Id: 5 000c50 08735770e\r\nFirmware Version: AR15\r\nUser Capacity:    8,001,563,222,016 bytes [8.00 TB]\r\nSector Sizes:     512 bytes logical, 4096 bytes physical\r\nRotation Rate:    5980 rpm\r\nDevice is:        Not in smartctl database [for details use: -P showall]\r\nATA Version is:   ACS-2, ACS-3 T13\/2161-D revision 3b\r\nSATA Version is:  SATA 3.1, 6.0 Gb\/s (current: 6.0 Gb\/s)\r\nLocal Time is:    Tue Oct 29 07:01:14 2019 CET\r\n\r\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\r\n  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       229844224\r\n  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0\r\n  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       43\r\n  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0\r\n  7 Seek_Error_Rate         0x000f   079   060   030    Pre-fail  Always       -       86831809\r\n  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       33791\r\n 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0\r\n 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       43\r\n183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0\r\n184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0\r\n187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0\r\n188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0\r\n189 High_Fly_Writes         0x003a   084   084   000    Old_age   Always       -       16\r\n190 Airflow_Temperature_Cel 0x0022   072   055   045    Old_age   Always       -       28 (Min\/Max 27\/36)\r\n191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0\r\n192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       28\r\n193 Load_Cycle_Count        0x0032   084   084   000    Old_age   Always       -       33889\r\n194 Temperature_Celsius     0x0022   028   045   000    Old_age   Always       -       28 (0 17 0 0 0)\r\n195 Hardware_ECC_Recovered  0x001a   119   099   000    Old_age   Always       -       229844224\r\n197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0\r\n198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0\r\n199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0\r\n240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       18326 (15 246 0)\r\n241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       21871056144\r\n242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3261163264\r\n<\/pre>\n<h1>SWAP<\/h1>\n<p>fallocate -l 100G \/home\/swapfile<br \/>\nmkswap \/home\/swapfile<br \/>\nswapon \/home\/swapfile<\/p>\n<h1>debian\/devuan vs BSD<\/h1>\n<p>Mein pers\u00f6nlicher Eindruck, wo einige Unterschiede sind.<\/p>\n<p>* Bei D\/D werden Log Datein regelm\u00e4\u00dfig gel\u00f6scht. Monatlich? Nach einem Reboot? Bei BSD ist die Grundeinstellung, dass auch nach Jahren Log Datein vorhanden sind. Gut, um l\u00e4ngerfristige Ver\u00e4nderungen am System festzustellen. Von z.B. Hardware (Bus Errors), Malware<\/p>\n<h1>Festplatten testen<\/h1>\n<p>Zum testen von Festplatten nutze ich den destructiven Test von dem Programm badblocks. Die Festplatte wird dabei komplett vier mal \u00fcberschrieben. Die SMART Werte der Festplatte speichere ich vor dem Testen mit badblocks ab und hinterher ein Vergleich zu machen.<br \/>\nDa das Programm badblocks im Falle von fehlerhaften Bl\u00f6cken die Blocknummer auf die Konsole ausgibt und man bei vielen Ausgaben die Fortschrittsanzeige nicht mehr sieht, kann diese Ausgabe mit der -o Option in eine Datei umgeleitet werden.<\/p>\n<p>Add -b 4096 for a larger blocksize for large (>= 6TB) devices.<\/p>\n<p>badblocks -swv -o badblocks.txt \/dev\/sdb<\/p>\n<p>Test with random data 131072*4096b = 512MB<br \/>\nbadblocks -svw -b 4096 -o badblocks.txt -c 131072 -t random \/dev\/sdb<\/p>\n","protected":false},"excerpt":{"rendered":"<p>LD_LIBRARY_PATH=\/home\/kater\/bin\/ las2txt64 -parse xyz -odir . -i 47605560_all_thin.las -keep_class 2 GIT Repair Permissions The repository isn't configured to be a shared repository (see core.sharedRepository in git help config). If the output of: git config core.sharedRepository is not group or true or 1 or some mask, try running: git config core.sharedRepository group cd \/path\/to\/repo.git chgrp -R [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-2422","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"_links":{"self":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/2422","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2422"}],"version-history":[{"count":85,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/2422\/revisions"}],"predecessor-version":[{"id":5346,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=\/wp\/v2\/posts\/2422\/revisions\/5346"}],"wp:attachment":[{"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2422"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2422"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/roboblog.fatal-fury.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2422"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}