http://linux-ip.net – Great online book about advanced networking issues.
http://lartc.org/ – Howto and man pages
http://linux-ip.net – Great online book about advanced networking issues.
http://lartc.org/ – Howto and man pages
Here are some handy commands that I used recently to debug a server performance issue:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
<strong>> sar</strong> Linux 2.4.9-34smp (some.server.net) 07/28/2004 12:00:00 AM CPU %user %nice %system %idle 12:10:00 AM all 12.29 2.00 4.55 81.16 12:20:00 AM all 4.14 0.16 2.58 93.13 12:30:00 AM all 4.46 0.02 2.79 92.74 12:40:01 AM all 3.68 0.04 2.36 93.92 12:50:00 AM all 3.78 0.03 2.47 93.72 01:00:00 AM all 3.89 0.10 2.42 93.59 01:10:00 AM all 6.65 0.62 2.70 90.03 01:20:00 AM all 5.56 0.31 2.85 91.27 01:30:00 AM all 4.04 0.02 2.40 93.53 01:40:00 AM all 4.41 0.58 2.92 92.09 01:50:01 AM all 5.63 0.73 3.13 90.51 02:00:00 AM all 3.58 0.03 2.19 94.19 02:10:00 AM all 8.19 0.03 2.39 89.39 02:20:00 AM all 4.84 0.00 2.90 92.25 02:30:00 AM all 6.77 0.59 2.86 89.78 02:40:01 AM all 4.04 0.02 2.99 92.94 02:50:00 AM all 4.60 0.03 2.83 92.55 03:00:00 AM all 5.67 0.29 2.83 91.21 03:10:00 AM all 4.58 0.13 2.99 92.29 03:20:00 AM all 8.69 0.80 3.46 87.05 03:30:01 AM all 5.32 0.22 3.10 91.36 03:40:01 AM all 4.85 0.06 3.23 91.85 03:50:00 AM all 3.73 0.00 2.72 93.55 04:00:00 AM all 3.40 0.02 2.27 94.31 04:10:00 AM all 40.16 0.09 8.79 50.96 04:20:00 AM all 44.44 0.04 10.07 45.45 04:30:00 AM all 32.32 1.06 10.09 56.53 04:40:00 AM all 38.10 0.04 7.05 54.81 04:50:01 AM all 32.80 0.02 8.07 59.12 05:00:02 AM all 32.89 2.06 13.31 51.74 05:10:00 AM all 42.18 0.49 9.52 47.81 05:20:01 AM all 53.17 2.62 9.76 34.44 05:30:00 AM all 18.56 6.41 16.87 58.15 05:40:00 AM all 38.36 0.04 9.80 51.80 05:50:00 AM all 26.03 0.24 9.35 64.38 06:00:00 AM all 30.65 0.97 9.82 58.56 06:10:01 AM all 44.41 0.07 9.51 46.00 06:20:00 AM all 25.11 0.14 12.42 62.34 06:30:00 AM all 37.65 0.02 12.12 50.21 Average: all 17.02 0.54 5.76 76.68 |
Also “sar -A” give more useful info.
For sar to work, you need a cronjob like this:
1 2 |
# run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib/sa/sa1 1 1 |
The cronjob can go in a file at /etc/cron.d/sysstat on Redhat and similar systems.
Another useful command is “vmstat”:
1 2 3 4 5 6 7 |
<strong>> vmstat 1</strong> procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 5 0 44856 3040 50504 273364 1 2 78 107 53 129 20 6 74 1 5 0 44856 3164 50552 274176 0 0 2288 1036 513 766 21 11 68 1 4 0 44856 3056 50556 273496 0 0 3128 116 443 507 7 6 87 2 3 0 44856 3236 50564 274800 0 0 3200 460 557 558 8 7 85 |
We’ll start by installing Squid:
1 2 3 4 5 6 7 8 9 10 11 |
> cd /root > wget http://www.squid-cache.org/Versions/v2/2.5/squid-2.5.STABLE3.tar.gz > tar -xvzf squid-2.5.STABLE3.tar.gz > cd squid-2.5.STABLE3 > ./configure > make > make install > cd /root > rm -rf squid-2.5.STABLE3 > mkdir archive > mv squid-2.5.STABLE3.tar.gz archive |
Squid is now installed at /usr/local/squid.
Notice that we use the “wget” command to download the software. Then we unpack and build the software. The build sequence is very typical for Linux packages – configure, make, make install.
We finish up by saving the distribution to /root/archive and removing the build
directory that is no longer needed.
Configuration for squid is pretty simple in the basic case. All configuration is
stored in a single file located at /usr/local/squid/etc/squid.conf.
The squid.conf file can be edited with your favoriate text editor. All though there
are hundreds of configuration options only a few are needed in a basic install. We made the following changes:
a. Find the line that looks like this (approx line 684):
1 |
#cache_dir ufs /usr/local/squid/var/cache 100 16 256 |
Remove the “#” comment from the begining of the line and set the desired cache size and location:
1 |
cache_dir ufs /usr/local/squid/var/cache 768 16 256 |
This defines a cache of not more then 768MB of disk storage.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 |
b. Find the line that looks like this (approx line 1760): # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS Below this line add the following lines: acl our_networks src 172.0.0.0/8 http_access allow our_networks acl my_network src 127.0.0.1 http_access allow my_network The first new line defines an access control list (acl) called "our_networks" which includes the listed address ranges. The second line tells squid to accept connections from the "our_networks" acl. The next two lines allow access through the loopback address. c. Find the line (approx line 1973): # cache_effective_user nobody Uncomment this line and change the username to squid. Then add a line to set the group: cache_effective_user squid cache_effective_group squid For a basic setup that is all that is required in the way of configuration. Next well prepare the cache directories and start the proxy server. Type: > chown --recursive squid.squid cache > chown --recursive squid.squid logs > /usr/local/squid/sbin/squid -z This command may take several minutes to complete. Now start the server with: > /usr/local/squid/sbin/squid You can verify that squid is running by checking the log files: > tail /var/log/messages Now we can add a service control script to /etc/init.d. A suitable script can be downloaded from Reliable Penguin: > wget http://www.reliablepenguin.com/clients/misc/simplesquidguard.rh9.squid.service > mv simplewquidguard.rh9.squid.service /etc/init.d/squid > chmod 755 /etc/init.d/squid And finally squid can be set to start on bootup: > cd /etc/rc.d/rc5.d > ln -s ../init.d/squid S25squid > ln -s ../init.d/squid K25squid Next we want to install SquidGuard but it requires the BerkeleyDB database library package so we'll install it first: > cd /root > wget http://www.sleepycat.com/update/snapshot/db-3.3.11.tar.gz > tar -xvzf db-3.3.11.tar.gz > cd db-3.3.11 > cd build_unix > ../dist/configure > make > make install > ldconfig > cd /root > rm -rf db-3.3.11 > mv db-3.3.11.tar.gz archive The build procedure for BerkeleyDB is a little unusual - notice the directory that we change into before running configure. Also notice that we are using a 3.x version of BerkeleyDB. The current release if 4.X but SquidGuard needs the older version. Now we can download and install SquidGuard: > wget http://ftp.teledanmark.no/pub/www/proxy/squidGuard/squidGuard-1.2.0.tar.gz > tar -xvzf squidGuard-1.2.0.tar.gz > cd squidGuard-1.2.0 > ./configure --with-db=/usr/local/BerkeleyDB.3.2 > make > make install > mkdir /usr/local/squidGuard > mkdir /usr/local/squidGuard/logs > cp samples/sample.conf /usr/local/squidGuard And now squidGuard is installed. Now lets configure squid to utilize squidGuard. We'll need to edit the squid.conf file at /usr/local/squid/squid.conf. Look for the line: # TAG: redirect_program Below this section add this line: redirect_program /usr/local/bin/squidGuard And restart squid: > service squid restart Now squid will load and use squidGuard. Next lets install the blacklists: > cd /usr/local/squidGuard > wget http://ftp.teledanmark.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz > tar -xvzf blacklists.tar.gz > chmod --recursive g+w blacklists > chown --recursive squid.squid blacklists The next step is to configure squidGuard. We'll start with a very basic setup and then expand it later using the webmin interface. Create a new file at /usr/local/squidGuard/squidGuard.conf with the following contents: logdir /usr/local/squidGuard/logs dbhome /usr/local/squidGuard/blacklists source localnet { ip 172.0.0.0/8 } source localhost { ip 127.0.0.1 } dest ads { domainlist ads/domains urllist ads/urls #expressionlist ads/expressions } dest audio-video { domainlist audio-video/domains urllist audio-video/urls #expressionlist audio-video/expressions } dest gambling { domainlist gambling/domains urllist gambling/urls #expressionlist gambling/expressions } dest mail { domainlist mail/domains #urllist mail/urls #expressionlist mail/expressions } dest proxy { domainlist proxy/domains urllist proxy/urls #expressionlist proxy/expressions } dest violence { domainlist violence/domains urllist violence/urls expressionlist violence/expressions } dest aggressive { domainlist aggressive/domains urllist aggressive/urls #expressionlist aggressive/expressions } dest drugs { domainlist drugs/domains urllist drugs/urls #expressionlist drugs/expressions } dest hacking { domainlist hacking/domains urllist hacking/urls #expressionlist hacking/expressions } dest porn { domainlist porn/domains urllist porn/urls expressionlist porn/expressions } dest warez { domainlist warez/domains urllist warez/urls #expressionlist warez/expressions } acl { localnet { pass !porn all redirect http://www.acme.net } localhost { pass !porn all redirect http://www.acme.net } default { pass !porn all redirect http://www.acme.net } } Save the file. Next let's set permissions: > cd /usr/local/squidGuard > chown --recursive squid.squid * Restart squid: > service squid restart Now squidGuard is running. Next lets install webmin. Webmin can be downloaded from: http://www.webmin.com We'll use the RPM to simplify the install. The install goes like this: > wget http://unc.dl.sourceforge.net/sourceforge/webadmin/webmin-1.100-1.noarch.rpm > rpm -U webmin-1.100-1.noarch.rpm That's about it. Webmin is now installed on the server and can be accessed on port 10000. The default user is "root" and the password is the same as the root password. Webmin includes a module for managing Squid by default so there is nothing to add. We just need to configure the module to know where our install of Squid is located. Follow these steps: 1. Login to Webmin 2. Click the Servers icon on the top menu bar. 3. Click the Squid Proxy Server icon. 4. Click the Module Config link under the top menu bar. 5. In the "System Configuration" section of the module config form set the following: a. Full path to squid config file ==> /usr/local/squid/etc/squid.conf b. Squid executable ==> /usr/local/squid/sbin/squid c. Full path to PID file ==> /usr/local/squid/var/log/squid.pid d. Full path to Squid cache directory ==> /usr/local/squid/var/cache e. Squid cachemgr.cgi executable ==> /usr/local/squid/libexec/cachemgr.cgi f. Full path to squid log directory ==> /usr/local/squid/var/logs 6. Click the save button at the bottom of the form. That does it. Now Webmin can be used to manage Squid. Next lets install a webmin module for managing SquidGuard. 1. Login to Webmin 2. Click the "Webmin" icon in the top menu bar. 3. Click the "Webmin Configuration" icon. 4. Click the "Webmin Modules" icon. 5. Complete the "Install Module" form as follows: a. Select "From FTP or HTTP URL" and enter the url: http://www.niemueller.de/webmin/modules/squidguard/squidguard-0.91.2.wbm.gz b. Select "Grant access only to users and groups" and enter "root". c. Press the "Install Module From File" button. Next to configure the module: 1. Login to Webmin 2. Click the "Servers" icon in the top menu bar. 3. Click the "SquidGuard" icon. 4. Click the "Module Config" link below the top menu bar. 5. Complete the form as follows: a. Full path to Squidguard configuration file ==> /usr/local/squidGuard/squidGuard.conf b. Full path to binary for SquidGuard ==> /usr/local/bin/squidGuard c. Full path to PID file of Squid ==> /usr/local/squid/var/log/squid.pid d. User the Squid runs as ==> squid e. Group that Squid runs as ==> squid f. Use database to extract user information ==> No g. Press the Save button. And that takes care of the squidGuard module for Webmain. |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 |
#!/bin/sh # # chkconfig: - 91 35 # description: Starts and stops the OpenLDAP slapd daemon # # pidfile: /usr/local/openldap-2.2.4/var/run/slapd.pid # config: /usr/local/openldap-2.2.4/etc/openldap/slapd.conf INSTALL_BASE=/usr/local/openldap-2.2.4 BDB_BASE=/usr/local/BerkeleyDB.4.2 # Source function library. if [ -f /etc/init.d/functions ] ; then . /etc/init.d/functions elif [ -f /etc/rc.d/init.d/functions ] ; then . /etc/rc.d/init.d/functions else exit 0 fi # Avoid using root's TMPDIR unset TMPDIR # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ ${NETWORKING} = "no" ] && exit 0 # Check that smb.conf exists. [ -f $INSTALL_BASE/etc/openldap/slapd.conf ] || exit 0 RETVAL=0 start() { action "Recovering slapd database: " $BDB_BASE/bin/db_recover -h $INSTALL_BASE/var/openldap-data action "Recovering slapd database (US): " $BDB_BASE/bin/db_recover -h $INSTALL_BASE/var/openldap-us-data echo -n $"Starting slapd: " daemon $INSTALL_BASE/libexec/slapd RETVAL=$? echo return $RETVAL } stop() { KIND="SMB" echo -n $"Shutting down slapd: " action "" kill -INT `cat $INSTALL_BASE/var/run/slapd.pid` RETVAL=$? echo return $RETVAL } restart() { stop start } reload() { echo -n $"Reloading slapd.conf file: " killproc slapd -HUP RETVAL=$? echo return $RETVAL } rhstatus() { status slapd } case "$1" in start) start ;; stop) stop ;; restart) restart ;; reload) reload ;; status) rhstatus ;; *) echo $"Usage: $0 {start|stop|restart|status}" exit 1 esac exit $? |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 |
Useful How-To's: http://unthought.net/Software-RAID.HOWTO/ (OLD) http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.html Where to get raidtools: > cd /root > mkdir raid > cd raid/ > wget http://people.redhat.com/mingo/raidtools/raidtools-1.00.3.tar.gz For software raid we need kernel > 2.4 with raid patches and the raid tools. To test the kernel (from the FAQ): If your system has RAID support, you should have a file called /proc/mdstat. Remember it, that file is your friend. If you do not have that file, maybe your kernel does not have RAID support. See what the contains, by doing a cat /proc/mdstat. It should tell you that you have the right RAID personality (eg. RAID mode) registered, and that no RAID devices are currently active. Software raid is configured through /etc/raidtab. Here's an example for raid 1: raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/sdb6 raid-disk 0 device /dev/sdc5 raid-disk 1 The mkraid command is used to initialize a new raid array: mkraid /dev/md0 Initial State --------------- Here's the initial setup: > df Filesystem 1k-blocks Used Available Use% Mounted on /dev/hda2 38456340 1462116 35040720 5% / /dev/hda1 23302 5976 16123 28% /boot none 515348 0 515348 0% /dev/shm /dev/hdc1 39516436 32828 37476280 1% /mnt/drive2 > fdisk /dev/hda Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 4998 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 3 24066 83 Linux /dev/hda2 4 4867 39070080 83 Linux /dev/hda3 4868 4998 1052257+ 82 Linux swap > fdisk /dev/hdc Command (m for help): p Disk /dev/hdc: 16 heads, 63 sectors, 79656 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdc1 * 1 79656 40146592+ 83 Linux Test Setup - Standard --------------------- 1. Remove old partitions on /dev/hdc: > fdisk /dev/hdc Command (m for help): p Disk /dev/hdc: 16 heads, 63 sectors, 79656 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdc1 * 1 79656 40146592+ 83 Linux Command (m for help): d Partition number (1-4): 1 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 2. Create 2 equal sized partitions for testing: > fdisk /dev/hdc Command (m for help): p Disk /dev/hdc: 16 heads, 63 sectors, 79656 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-79656, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-79656, default 79656): +1024M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (2082-79656, default 2082): Using default value 2082 Last cylinder or +size or +sizeM or +sizeK (2082-79656, default 79656): +1024M Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 3. Setup /etc/raidtab: raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdc1 raid-disk 0 device /dev/hdc2 raid-disk 1 4. Prepare the partitions with mkraid: > mkraid /dev/md0 handling MD device /dev/md0 analyzing super-block disk 0: /dev/hdc1, 1048792kB, raid superblock at 1048704kB disk 1: /dev/hdc2, 1048824kB, raid superblock at 1048704kB > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hdc2[1] hdc1[0] 1048704 blocks [2/2] [UU] [========>............] resync = 44.2% (465664/1048704) finish=0.9min speed=10171K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hdc2[1] hdc1[0] 1048704 blocks [2/2] [UU] [=========>...........] resync = 48.7% (512512/1048704) finish=0.8min speed=10056K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hdc2[1] hdc1[0] 1048704 blocks [2/2] [UU] [===============>.....] resync = 79.7% (836608/1048704) finish=0.3min speed=10148K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hdc2[1] hdc1[0] 1048704 blocks [2/2] [UU] unused devices: <none> 5. Checkout the results: > fdisk /dev/md0 Command (m for help): p Disk /dev/md0: 2 heads, 4 sectors, 262176 cylinders Units = cylinders of 8 * 512 bytes Device Boot Start End Blocks Id System /dev/md0p1 8 10036530 40146088+ 83 Linux Partition 1 does not end on cylinder boundary: phys=(1023, 15, 63) should be (1023, 1, 4) Command (m for help): 6. Mount up the new partition: > mkdir /mnt/raid > mount /dev/md0 /mnt/raid > df Filesystem 1k-blocks Used Available Use% Mounted on /dev/hda2 38456340 1462376 35040460 5% / /dev/hda1 23302 5976 16123 28% /boot none 515348 0 515348 0% /dev/shm /dev/md0 39516436 32828 37476280 1% /mnt/raid 7. Shutdown raid: > umount /dev/md0 > raidstop --all /dev/md0 Test Setup - Degraded Mode -------------------------- 1. Create partitions: > fdisk /dev/hdc Command (m for help): p Disk /dev/hdc: 16 heads, 63 sectors, 79656 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdc1 1 2081 1048792+ 83 Linux /dev/hdc2 2082 4162 1048824 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (4163-79656, default 4163): Using default value 4163 Last cylinder or +size or +sizeM or +sizeK (4163-79656, default 79656): +1024M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 4 First cylinder (6244-79656, default 6244): Using default value 6244 Last cylinder or +size or +sizeM or +sizeK (6244-79656, default 79656): +1024M Command (m for help): p Disk /dev/hdc: 16 heads, 63 sectors, 79656 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdc1 1 2081 1048792+ 83 Linux /dev/hdc2 2082 4162 1048824 83 Linux /dev/hdc3 4163 6243 1048824 83 Linux /dev/hdc4 6244 8324 1048824 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 2. Format/mount the first new partition. > mke2fs /dev/hdc3 mke2fs 1.27 (8-Mar-2002) warning: 62 blocks unused. Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 131328 inodes, 262144 blocks 13110 blocks (5.00%) reserved for the super user First data block=0 8 block groups 32768 blocks per group, 32768 fragments per group 16416 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 30 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. > mount /dev/hdc3 /mnt/drive2 3. Setup /etc/raidtab: raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdc1 raid-disk 0 device /dev/hdc2 raid-disk 1 raiddev /dev/md1 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdc4 raid-disk 0 device /dev/hdc3 failed-disk 1 4. Build the array: > mkraid /dev/md1 handling MD device /dev/md1 analyzing super-block disk 0: /dev/hdc4, 1048824kB, raid superblock at 1048704kB disk 1: /dev/hdc3, failed > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hdc4[0] 1048704 blocks [2/1] [U_] unused devices: <none> 5. Put a filesystem on the array: > mke2fs /dev/md1 mke2fs 1.27 (8-Mar-2002) warning: 32 blocks unused. Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 131328 inodes, 262144 blocks 13108 blocks (5.00%) reserved for the super user First data block=0 8 block groups 32768 blocks per group, 32768 fragments per group 16416 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 23 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. 6. Copy contents of failed drive to array: > mount /dev/md1 /mnt/raid/ > cd /mnt/drive2 > find . -xdev | cpio -pm /mnt/raid > umount /dev/hdc3 7. Edit /etc/raidtab making failed drive active: raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdc1 raid-disk 0 device /dev/hdc2 raid-disk 1 raiddev /dev/md1 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdc4 raid-disk 0 device /dev/hdc3 raid-disk 1 8. Hot add the failed drive to the array: > raidhotadd /dev/md1 /dev/hdc3 > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hdc3[2] hdc4[0] 1048704 blocks [2/1] [U_] [=>...................] recovery = 8.4% (90048/1048704) finish=1.5min speed=10005K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hdc3[2] hdc4[0] 1048704 blocks [2/1] [U_] [==========>..........] recovery = 51.8% (544640/1048704) finish=0.8min speed=10131K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hdc3[2] hdc4[0] 1048704 blocks [2/1] [U_] [===============>.....] recovery = 77.7% (816512/1048704) finish=0.4min speed=9480K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md1 : active raid1 hdc3[1] hdc4[0] 1048704 blocks [2/2] [UU] unused devices: <none> Thats it! Test Setup - Degraded Mode - Real Drives ---------------------------------------- 1. Check initial config of primary drive. > fdisk /dev/hda Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 1870 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 3 24066 83 Linux /dev/hda2 4 41 305235 82 Linux swap /dev/hda3 42 1870 14691442+ 83 Linux 2. Partition secondary drive. > fdisk /dev/hdd Command (m for help): p Disk /dev/hdd: 16 heads, 63 sectors, 29805 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-29805, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-29805, default 29805): +25M Command (m for help): p Disk /dev/hdd: 16 heads, 63 sectors, 29805 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdd1 1 51 25672+ 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (52-29805, default 52): Using default value 52 Last cylinder or +size or +sizeM or +sizeK (52-29805, default 29805): +300M Command (m for help): p Disk /dev/hdd: 16 heads, 63 sectors, 29805 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdd1 1 51 25672+ 83 Linux /dev/hdd2 52 661 307440 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (662-29805, default 662): Using default value 662 Last cylinder or +size or +sizeM or +sizeK (662-29805, default 29805): Using default value 29805 Command (m for help): p Disk /dev/hdd: 16 heads, 63 sectors, 29805 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdd1 1 51 25672+ 83 Linux /dev/hdd2 52 661 307440 83 Linux /dev/hdd3 662 29805 14688576 83 Linux Command (m for help): p Disk /dev/hdd: 16 heads, 63 sectors, 29805 cylinders Units = cylinders of 1008 * 512 bytes Device Boot Start End Blocks Id System /dev/hdd1 * 1 51 25672+ 83 Linux /dev/hdd2 52 661 307440 83 Linux /dev/hdd3 662 29805 14688576 83 Linux Command (m for help): a Partition number (1-4): 1 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 3. Make a filesystem on the /boot partition of the secondary drive and mount it. > mke2fs /dev/hdd1 mke2fs 1.27 (8-Mar-2002) Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) 6432 inodes, 25672 blocks 1283 blocks (5.00%) reserved for the super user First data block=1 4 block groups 8192 blocks per group, 8192 fragments per group 1608 inodes per group Superblock backups stored on blocks: 8193, 24577 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 32 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. > mount /dev/hdd1 /mnt/drive2 4. Install grub on the /boot partition of the secondary drive. > grub-install --root-directory=/mnt/drive2 /dev/hdd Probing devices to guess BIOS drives. This may take a long time. Installation finished. No error reported. This is the contents of the device map /mnt/drive2/boot/grub/device.map. Check if this is correct or not. If any of the lines is incorrect, fix it and re-run the script `grub-install'. (fd0) /dev/fd0 (hd0) /dev/hda (hd1) /dev/hdd > cd /mnt/drive2 > ln -s ./boot/grub grub 5. Copy kernel files to /boot partition of the secondary drive. > cp /boot/* /mnt/drive2/ > cp /boot/grub/grub.conf /mnt/drive2/grub > cd /mnt/drive2/grub > ln -s ./grub.conf ./boot/grub/menu.lst > cd .. > umount /dev/hdd1 6. Make the swap partition on the secondary drive. > mkswap /dev/hdd2 Setting up swapspace version 1, size = 307436K 7. Setup the /etc/raidtab file in degraded mode: raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdd3 raid-disk 0 device /dev/hda3 failed-disk 1 8. Build the array: > mkraid /dev/md0 handling MD device /dev/md0 analyzing super-block disk 0: /dev/hdd3, 14688576kB, raid superblock at 14688512kB disk 1: /dev/hda3, failed 9. Put ext3 filesystem on the array: > mkfs -t ext3 /dev/md0 mke2fs 1.27 (8-Mar-2002) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 1836928 inodes, 3672128 blocks 183606 blocks (5.00%) reserved for the super user First data block=0 113 block groups 32768 blocks per group, 32768 fragments per group 16256 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 37 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. 10. Copy contents of failed drive to array: > mkdir /mnt/raid > mount /dev/md0 /mnt/raid/ > cd / > find . -xdev | cpio -pm /mnt/raid 11. Create the initrd images: > mkinitrd --preload raid1 --with=raid1 initrd-2.4.18-3.img 2.4.18-3 > cp initrd-2.4.18-3.img /mnt/drive2/ > cp /boot/initrd-2.4.18-3.img /boot/initrd-2.4.18-3.img.bak > cp initrd-2.4.18-3.img /boot 12. Set partition type on secondary drive: > fdisk /dev/hdd Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): fd Changed system type of partition 3 to fd (Linux raid autodetect) Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot. Syncing disks. 13. Edit /etc/fstab on the array: Change the root (/) line to absolute instead of label reference. 14. REBOOT 15. Edit /etc/raidtab and change failed drive to active. raiddev /dev/md0 raid-level 1 nr-raid-disks 2 nr-spare-disks 0 chunk-size 4 persistent-superblock 1 device /dev/hdd3 raid-disk 0 device /dev/hda3 raid-disk 1 16. Charge partition type of failed drive using fdisk. > fdisk /dev/hda The number of cylinders for this disk is set to 1870. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 1870 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 3 24066 83 Linux /dev/hda2 4 41 305235 82 Linux swap /dev/hda3 42 1870 14691442+ 83 Linux Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): fd Changed system type of partition 3 to fd (Linux raid autodetect) Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot. Syncing disks. 17. REBOOT 18. Hot add the failed drive > raidhotadd /dev/md0 /dev/hda3 > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hda3[2] hdd3[0] 14688512 blocks [2/1] [U_] [>....................] recovery = 0.3% (48704/14688512) finish=161.6min speed=1509K/sec unused devices: <none> > cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 hda3[1] hdd3[0] 14688512 blocks [2/2] [UU] unused devices: <none> DONE !!! Boot Considerations ------------------- Current Future ------- --------- /dev/hda1 -> grub boot partition -> /dev/hda1 or /dev/hdc1 /dev/hda2 -> root partition -> /dev/md0 /dev/hdc1 -> unused /dev/hdc2 -> unused /dev/md0 = /dev/hda2 + /dev/hdc2 (RAID 1 Mirror) Issue 1: Raid is provided as modules but is needed before the root filesystem is booted. mkinitrd --with=<module> <ramdisk name> <kernel> mkinitrd --preload raid5 --with=raid5 raid-ramdisk 2.2.5-22 An initrd command must be setup in the grub bootloader for this to work. Issue 2: The bios is set to boot from /dev/hda1. How do we get booted from /dev/hdc1 with the root set to /dev/md0? Solution: The search order for bootable drives in the BIOS should be set to /dev/hda and then /dev/hdc. This may be automatic. Current GRUB Config # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/hda2 # initrd /initrd-version.img #boot=/dev/hda default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz title Red Hat Linux (2.4.20-13.7) root (hd0,0) kernel /vmlinuz-2.4.20-13.7 ro root=/dev/hda2 initrd /initrd-2.4.20-13.7.img New GRUB Config # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/hda2 # initrd /initrd-version.img #boot=/dev/hda default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz title Red Hat Linux (2.4.20-13.7) Primary root (hd0,0) kernel /vmlinuz-2.4.20-13.7 ro root=/dev/md0 initrd /initrd-2.4.20-13.7.img title Red Hat Linux (2.4.20-13.7) Backup root (hd1,0) kernel /vmlinuz-2.4.20-13.7 ro root=/dev/md0 initrd /initrd-2.4.20-13.7.img Setup Plan ---------- 1. Configure the /etc/raidtab for a degraded raid 1 mirror using /dev/hda2 and /dev/hdc2 with /dev/hdc2 as the active device and /dev/hda2 as a failed device. 2. Create new initrd with raid1 support. Reboot and verify that new initrd is working. 2. Install modified grub on /dev/hdc1 and /dev/hda1. 3. Copy contents of hda2 to hdc2. 5. Reboot and verify that md0/hdc2 is working. 6. Hot add hda2 to the md0 array. And that should do the trick! |