rc.local won't run

[NewtownGuy]

Hello,

I must have messed something up. Maybe I turned off some system service that I need ?

/etc/rc.d/rc.local on Fedora Core 3 (2.6.9-1.667) won't run at all on a new machine. It has 755 permission, after being loaded from another machine. I've used the same file on other machines of the same distro, and all is well. Linux boots and runs without errors. I know rc.local is not running because it has a line to pipe dmesg to a file, and the file is not being created.

What log file records all of the steps during booting, including the running of rc.local ? I looked in boot.log on other machines, and I don't see anything about the execution of rc.local.

What would prevent rc.local from running, or what should invoke it but maybe isn't, and how do I fix it ?

Thank you in advance for your help.

-- NewtownGuy

 

[Annihilannic]

You'll probably find that it's symbolically linked from the /etc/rc?.d directories, e.g.

[tt]# ls -l /etc/rc?.d/*local
lrwxrwxrwx 1 root root 11 Dec 27 11:38 /etc/rc2.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Dec 27 11:38 /etc/rc3.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Dec 27 11:38 /etc/rc4.d/S99local -> ../rc.local
lrwxrwxrwx 1 root root 11 Dec 27 11:38 /etc/rc5.d/S99local -> ../rc.local
[/tt]

Usually the first command in that script is:

[tt]touch /var/lock/subsys/local[/tt]

Is that file being touched when you boot?

As far as I know any messages displayed by this script should be recorded in /var/log/boot.log as you expected.

Are you aware that dmesg output is already dumped to /var/log/dmesg during the system boot?

Annihilannic.

 

[NewtownGuy]

To: Annihilannic

Thank you for your reply.

I do not see the "local" file in the folder you mentioned. Where else should I look ? Must the "touch" line be first in rc.local ? At the moment, it's the last.

I've looked in boot.log, and I don't see any indication that rc.local is being run. I've tried putting an "echo" statement in rc.local, but the echo text is not displayed.

I'm trying to write dmesg to my own file because I understand there's a limit on the size of a circular buffer that is maintained for it otherwise. Is this correct ?

-- NewtownGuy

 

[Annihilannic]

Yes, there is a limit on the size of the kernel ring buffer, however dumping it to your own file will not circumvent that. If your kernel displays too many messages during the boot process and fills that buffer, it will happen before your scripts run and have a chance to save the output. The only advantage you may gain is saving copies of that output for multiple reboots by appending it to a file rather than overwriting it, which is what happens to the /var/log/dmesg file.

The symbolic links I quoted are called S99local, not local, are you sure you were looking for the right file names?

I think the touch line being at the end of the file is fine, it just means that if the script "errors out" before it gets to that line, it may not be executed. Which you may argue is correct since it shouldn't lock that subsystem unless it was successfully "started".

Annihilannic.

 

[NewtownGuy]

To: Annihilannic

Here's the link listing you requested from my machine:

[root@B-2 ~]# ls -l /etc/rc*.d/*local
lrwxrwxrwx  1 root root  11 Feb 15 16:01 /etc/rc2.d/S99local -> ../rc.local
lrwxrwxrwx  1 root root  11 Feb 15 16:01 /etc/rc3.d/S99local -> ../rc.local
lrwxrwxrwx  1 root root  11 Feb 15 16:01 /etc/rc4.d/S99local -> ../rc.local
lrwxrwxrwx  1 root root  11 Feb 15 16:01 /etc/rc5.d/S99local -> ../rc.local
-rwxr-xr-x  1 root root 558 Feb 15 20:08 /etc/rc.d/rc.local

This is consistent with the links you described. Why won't rc.local run, and why doesn't it run multiple times since it's referenced four times above ?

-- NewtownGuy

 

[SamBones]

You should also have a link in [tt]/etc[/tt] called [tt]rc.local[/tt]...

# ls -la /etc/rc.local
lrwxrwxrwx  1 root root 13 Dec 16 20:49 /etc/rc.local -> rc.d/rc.local

 

[NewtownGuy]

To: SamBones

I'm still stumped. Here's a full listing for rc*, including links and files. It seems everything is here. I triple-checked /etc/rc.d/rc.local, and put an echo line as the first executable line, but it just doesn't run. Is there something in setup that I might have turned off that I need ?

[root@B-2 etc]# ls rc* -all
lrwxrwxrwx   1 root root    7 Feb 15 16:01 rc -> rc.d/rc
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc0.d -> rc.d/rc0.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc1.d -> rc.d/rc1.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc2.d -> rc.d/rc2.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc3.d -> rc.d/rc3.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc4.d -> rc.d/rc4.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc5.d -> rc.d/rc5.d
lrwxrwxrwx   1 root root   10 Feb 15 16:01 rc6.d -> rc.d/rc6.d
lrwxrwxrwx   1 root root   13 Feb 15 16:01 rc.local -> rc.d/rc.local
lrwxrwxrwx   1 root root   15 Feb 15 16:01 rc.sysinit -> rc.d/rc.sysinit

rc.d:
total 132
drwxr-xr-x  10 root root  4096 Feb 18 11:14 .
drwxr-xr-x  71 root root 12288 Feb 18 04:02 ..
drwxr-xr-x   2 root root  4096 Feb 15 16:04 init.d
-rwxr-xr-x   1 root root  2352 Mar 16  2004 rc
drwxr-xr-x   2 root root  4096 Feb 15 16:04 rc0.d
drwxr-xr-x   2 root root  4096 Feb 15 16:04 rc1.d
drwxr-xr-x   2 root root  4096 Feb 15 16:04 rc2.d
drwxr-xr-x   2 root root  4096 Feb 15 17:57 rc3.d
drwxr-xr-x   2 root root  4096 Feb 15 16:05 rc4.d
drwxr-xr-x   2 root root  4096 Feb 15 16:05 rc5.d
drwxr-xr-x   2 root root  4096 Feb 15 16:04 rc6.d
-rwxr-xr-x   1 root root   558 Feb 15 20:08 rc.local
-rwxr-xr-x   1 root root 26826 Oct 17  2004 rc.sysinit
[root@B-2 etc]#

-- NewtownGuy

 

[Annihilannic]

Why won't rc.local run, and why doesn't it run multiple times since it's referenced four times above ?

These scripts are only run when the system enters that specific runlevel, so they should only be run once. They are run by the /etc/rc.d/rc script, which also includes a check for that file in /var/lock/subsys to prevent it from attempting to start services that are already running.

I can't see any obvious reason why it isn't being run. Maybe you should take a backup of /etc/rc.d/rc and modify it to add some extra debugging to the loop where it runs the start scripts?

What runlevel are you booting into by default? (who -r will tell you the current one, and the 'initdefault' line in /etc/inittab will tell you the default).

Annihilannic.

 

[NewtownGuy]

To: Annihilannic

I'm at run-level 3.

I did a clean install on a new hard drive, and rc.local was clearly running because the "touch" file was present and had a date and time code of when the system booted. I then edited rc.local, adding dmesg >> file, and now it has stopped working again. The "touch" file is no longer present. I tried restoring the original rc.local file, but it still won't run.

Something is getting confused and I can't make it right again. Any suggestions ??? I don't know enough about the workings of rc or rc.sysinit to try to debug them.

Thank you.

-- NewtownGuy

 

[elgrandeperro]

Are you running SELinux security?

When things don't run and they look like they should, I have found the problems in extended ACLS (in solaris) or SELinux.

If so, check with ls -lZ. I just copied my rc.local to new and you can see the difference:

-rwxr-xr-x root root user_ubject_r:etc_t new
-rwxr-xr-x root root system_ubject_r:initrc_exec_t rc.local

If SELinux were set to "Enforcing" then the perms for "new" would not be runnable for init, much like when you

eugene

 

[NewtownGuy]

No, I turned it off during installation. When I run ls -lZ, I get, "Sorry, this option can only be used on a SELinux kernel."

However, the behavior -- blocking my rc.local -- is the same as you describe.

What debug code should I put into rc (or whatever) to check when rc.local is supposed to be run, and to see why it is not being run ? Where should I put it in the file ?

-- NewtownGuy

 

[Annihilannic]

You could try putting a set -x in /etc/rc.d/rc on the line before:

[tt]...
# Now run the START scripts.
for i in /etc/rc$runlevel.d/S* ; do
...[/tt]

I'm presuming it's similar to the /etc/rc.d/rc script on Red Hat Enterprise Linux AS4 which is the most similar operating system I currently have access to. Make sure you back it up first, ideally using cp -p /etc/rc.d/rc /etc/rc.d/rc.backup or similar to preserve timestamps, etc. on the original file.

Annihilannic.

 

[NewtownGuy]

Yes, the text you gave is identical to rc on my machine.

I added set -x, and now I get a lot of extra text during booting, but it goes by too fast to read. It's not in dmesg or boot.log. What file has this text in it ?

-- NewtownGuy

 

[elgrandeperro]

Things I would check:

That you didn't accidently change the first line:
#!/bin/sh
or
#!/bin/sh -x (to see debug)

That there isn't a control character in the file:
od -c rc.local

The file says it is okay:

# file rc.local
rc.local: Bourne shell script text executable

 

[Annihilannic]

Hmm... I was hoping it would go to boot.log.

Try adding this before the set -x:

[tt]exec > /tmp/rc.out 2>&1[/tt]

The output should be in that file, you probably won't see any other boot messages on the console after that point.

Annihilannic.

 

[NewtownGuy]

To: elgrandeperro

I get a different format output from that command, but I don't see any ctl-C's here.

NOTE: I'm composing the file on Windows and transferring it via SSH Secure File Transfer to the Linux machine. Is that causing problems with CR and LF ?

[root@B-3 rc.d]# od -c rc.local
0000000   #   !   /   b   i   n   /   s   h  \r  \n   #  \r  \n   #
0000020   T   h   i   s       s   c   r   i   p   t       w   i   l   l
0000040       b   e       e   x   e   c   u   t   e   d       *   a   f
0000060   t   e   r   *       a   l   l       t   h   e       o   t   h
0000100   e   r       i   n   i   t       s   c   r   i   p   t   s   .
0000120  \r  \n   #       Y   o   u       c   a   n       p   u   t
0000140   y   o   u   r       o   w   n       i   n   i   t   i   a   l
0000160   i   z   a   t   i   o   n       s   t   u   f   f       i   n
0000200       h   e   r   e       i   f       y   o   u       d   o   n
0000220   '   t  \r  \n   #       w   a   n   t       t   o       d   o
0000240       t   h   e       f   u   l   l       S   y   s       V
0000260   s   t   y   l   e       i   n   i   t       s   t   u   f   f
0000300   .  \r  \n  \r  \n  \r  \n   t   o   u   c   h       /   v   a
0000320   r   /   l   o   c   k   /   s   u   b   s   y   s   /   l   o
0000340   c   a   l  \r  \n
0000345
[root@B-3 rc.d]#

-- NewtownGuy

 

[Annihilannic]

Yes, that's a bad thing. If you really must edit the file in Windows, try and use an editor that recognises the Unix text format (with only LFs as line terminators), or remember to convert it back to Unix format on the destionation system using tr -d '\r' < windowsfile > unixfile, or dos2unix < windowsfile > unixfile if available.

Annihilannic.

 

[NewtownGuy]

To: Annihilannic

Here's a small part of the log file that may help:

Starting atd: [  OK  ]
+ for i in '/etc/rc$runlevel.d/S*'
+ check_runlevel /etc/rc3.d/S99local
+ '[' -x /etc/rc3.d/S99local ']'
+ case "$1" in
+ return 0
+ subsys=local
+ '[' -f /var/lock/subsys/local -o -f /var/lock/subsys/local.init ']'
+ '[' -f /var/run/confirm ']'
+ update_boot_stage local
+ '[' yes = yes -a -x /usr/bin/rhgb-client ']'
+ return 0
+ '[' local = halt -o local = reboot ']'
+ egrep -q '(daemon |action |success |failure )' /etc/rc3.d/S99local
+ '[' local = single -o local = local ']'
+ /etc/rc3.d/S99local start
[COLOR=red]/etc/rc.d/rc: /etc/rc3.d/S99local: /bin/sh
: bad interpreter: No such file or directory[/color]
+ rm -f /var/run/confirm
+ '[' -x /usr/bin/rhgb-client ']'

There are several sym links that finally get me to /etc/rc.d/rc.local. Would an extra LF or CR cause the "bad interpreter" error message above, and be the cause of all of this ?

-- NewtownGuy

 

[Annihilannic]

Yes, I would say that's very likely your problem, note the similar output in my test:

[tt]$ echo '#!/bin/sh
>
> date' > testscript
$ chmod 755 testscript
$ ./testscript
Wed Feb 20 01:34:22 GMT 2008
$ unix2dos < testscript > testscript.dos
$ chmod 755 testscript.dos
$ ./testscript.dos
: bad interpreter: No such file or directory
$[/tt]

Is there any reason why you are editing these files on Windows? If it's because of the steep vi learning curve, try an alternative such as joe perhaps?

Annihilannic.

 

[NewtownGuy]

FIXED !!!! THANKS TO ALL !!!

The reason that I'm doing editing on Windows and transferring the updated text files to Linux is that I need to make changes in a set of configuration, HTML and other files, for each of a number of similar machines. I keep all master copies, and do all the editing on, a Windows machine.

How can I do this without having to dos2unix every single text file that I transfer from Windows to Linux ? In some cases, I also need to transfer the original files from Linux to Windows so I can edit them and keep master copies, before moving them back to the Linux machines.

-- NewtownGuy