From JRizzo at ea.com Thu Dec 5 23:19:18 2002 From: JRizzo at ea.com (Rizzo, Joe) Date: Thu, 5 Dec 2002 15:19:18 -0800 Subject: clogin and config mode Message-ID: <194487E5B6AC1E4E82FCBD56E5799791BC65F5@eahq-mb3.rws.ad.ea.com> When using the -c or -x options of clogin to script config changes, I am unable to enter config mode and enter commands. I have tried on a variety of Cisco models and IOS versions with no luck. Has anyone else seen this? Thanks, Joe Example: $clogin -c "conf t; exit" router router spawn telnet router Connected to router. Escape character is '^]'. User Access Verification Password: router>enable Password: router# router#term length 0 router#conf t Enter configuration commands, one per line. End with CNTL/Z. *** Here is where it hangs *** From asp at partan.com Thu Dec 5 23:52:02 2002 From: asp at partan.com (Andrew Partan) Date: Thu, 5 Dec 2002 18:52:02 -0500 Subject: clogin and config mode In-Reply-To: <194487E5B6AC1E4E82FCBD56E5799791BC65F5@eahq-mb3.rws.ad.ea.com> References: <194487E5B6AC1E4E82FCBD56E5799791BC65F5@eahq-mb3.rws.ad.ea.com> Message-ID: <20021205235202.GA98823@partan.com> On Thu, Dec 05, 2002 at 03:19:18PM -0800, Rizzo, Joe wrote: > When using the -c or -x options of clogin to script config changes, I am > unable to enter config mode and enter commands. I have tried on a > variety of Cisco models and IOS versions with no luck. Prompt recognition. $prompt is probably set to 'router#' - which does not match the prompt in config mode - router(config)# You want to do something like: clogin -c "conf t\rip name-server 198.6.255.248\rend;write" router - basically glom all of the config commands together with \r between them until you get all the way out of config mode. Or set up a per router tftp/rcp// file on your server and use something like util/cisco-load.exp to load it. --asp From AZhang at reliant.com Mon Dec 23 20:58:26 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Mon, 23 Dec 2002 14:58:26 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6EC@rriexmb10.services.reinternal.com> SunOS netadmin 5.8 Generic_108528-07 sun4u sparc SUNW,UltraSPARC-IIi-cEngine rancid-2.2.2, expect-5.38, tcl-8.4.1, tk-8.4.1 openssh-3.4p1 Rancid over telnet runs well. Rancid over ssh hangs, clogin -c "show run" hangs, clogin and then "show run" at the enable prompt also hangs. Below is the tail portion of clogin -c "show run" ecdc2ibgp with expect -d. Pointers will be appreciated. expect: does "ntp server 158.81.250.130\r\nend\r\n\r\necdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? no "^[^\n\r]*ecdc2ibgp(\([^\r\n]+\))?#."? no "[\n\r]+"? yes expect: set expect_out(0,string) "\r\n" expect: set expect_out(spawn_id) "exp4" expect: set expect_out(buffer) "ntp server 158.81.250.130\r\n" ntp server 158.81.250.130 expect: continuing expect expect: does "end\r\n\r\necdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? no "^[^\n\r]*ecdc2ibgp(\([^\r\n]+\))?#."? no "[\n\r]+"? yes expect: set expect_out(0,string) "\r\n\r\n" expect: set expect_out(spawn_id) "exp4" expect: set expect_out(buffer) "end\r\n\r\n" end expect: continuing expect expect: does "ecdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? yes expect: set expect_out(0,string) "ecdc2ibgp#" expect: set expect_out(spawn_id) "exp4" expect: set expect_out(buffer) "ecdc2ibgp#" ecdc2ibgp#send: sending "exit\r" to { exp4 } expect: does "Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match glob pattern "Do you wish to save your configuration changes"? no "\n"? yes expect: set expect_out(0,string) "\n" expect: set expect_out(spawn_id) "exp4" expect: set expect_out(buffer) "Received disconnect from 158.81.248.251: Time-out activated\r\n" expect: continuing expect expect: does "" (spawn_id exp4) match glob pattern "Do you wish to save your configuration changes"? no "\n"? no expect: read eof expect: set expect_out(spawn_id) "exp4" expect: set expect_out(buffer) "" From heas at shrubbery.net Mon Dec 23 22:39:28 2002 From: heas at shrubbery.net (john heasley) Date: Mon, 23 Dec 2002 14:39:28 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E3504C6EC@rriexmb10.services.reinternal.com>; from AZhang@reliant.com on Mon, Dec 23, 2002 at 02:58:26PM -0600 References: <4542F75EC5DC2E44AA0B648E20D00E3504C6EC@rriexmb10.services.reinternal.com> Message-ID: <20021223143928.H29493@shrubbery.net> are you saying that after the last line of the debug output below, you do not receive a shell prompt? or, are you referring to the 'time-out activated' that appears in the output? if the latter, i suspect it is the cisco that is disconnecting and the fix would be to increase your vty session and/or exec timeouts. the cisco does not reset it's timer on output, only input. it is conceivable that it could take long enough to generate and display the configuration to activate the timer. if the former, maybe the expect_before is not remaining active. i would have to research that, since it works here (sparc, netbsd, exp 5.33, tcl 8.3.2). Mon, Dec 23, 2002 at 02:58:26PM -0600, Zhang, Anchi: > SunOS netadmin 5.8 Generic_108528-07 sun4u sparc SUNW,UltraSPARC-IIi-cEngine > rancid-2.2.2, expect-5.38, tcl-8.4.1, tk-8.4.1 > openssh-3.4p1 > > Rancid over telnet runs well. > > Rancid over ssh hangs, clogin -c "show run" hangs, clogin and then "show run" at the enable prompt also hangs. Below is the tail portion of clogin -c "show run" ecdc2ibgp with expect -d. Pointers will be appreciated. > > expect: does "ntp server 158.81.250.130\r\nend\r\n\r\necdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no > "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? no > "^[^\n\r]*ecdc2ibgp(\([^\r\n]+\))?#."? no > "[\n\r]+"? yes > expect: set expect_out(0,string) "\r\n" > expect: set expect_out(spawn_id) "exp4" > expect: set expect_out(buffer) "ntp server 158.81.250.130\r\n" > ntp server 158.81.250.130 > expect: continuing expect > > expect: does "end\r\n\r\necdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no > "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? no > "^[^\n\r]*ecdc2ibgp(\([^\r\n]+\))?#."? no > "[\n\r]+"? yes > expect: set expect_out(0,string) "\r\n\r\n" > expect: set expect_out(spawn_id) "exp4" > expect: set expect_out(buffer) "end\r\n\r\n" > end > > expect: continuing expect > > expect: does "ecdc2ibgp#Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match regular expression "\u0008+"? no > "^[^\n\r *]*ecdc2ibgp(\([^\r\n]+\))?#"? yes > expect: set expect_out(0,string) "ecdc2ibgp#" > expect: set expect_out(spawn_id) "exp4" > expect: set expect_out(buffer) "ecdc2ibgp#" > ecdc2ibgp#send: sending "exit\r" to { exp4 } > > expect: does "Received disconnect from 158.81.248.251: Time-out activated\r\n" (spawn_id exp4) match glob pattern "Do you wish to save your configuration changes"? no > "\n"? yes > expect: set expect_out(0,string) "\n" > expect: set expect_out(spawn_id) "exp4" > expect: set expect_out(buffer) "Received disconnect from 158.81.248.251: Time-out activated\r\n" > expect: continuing expect > > expect: does "" (spawn_id exp4) match glob pattern "Do you wish to save your configuration changes"? no > "\n"? no > expect: read eof > expect: set expect_out(spawn_id) "exp4" > expect: set expect_out(buffer) "" From AZhang at reliant.com Mon Dec 23 23:00:29 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Mon, 23 Dec 2002 17:00:29 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6EE@rriexmb10.services.reinternal.com> I do get the shell prompt back right after the last line of the debug but it takes a long time to get there. Rancid over ssh does work with some of my routers but not all. For example, it had worked with one router until I added a few more lines to its existing ACL. The strange thing is that if I login using clogin and issue "show run" at the router's command prompt, the display will be fine. However, if I do "term len 0" and then "show run" the display hangs when it gets close to the very end of the config. Anchi From heas at shrubbery.net Mon Dec 23 23:15:54 2002 From: heas at shrubbery.net (john heasley) Date: Mon, 23 Dec 2002 15:15:54 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E3504C6EE@rriexmb10.services.reinternal.com>; from AZhang@reliant.com on Mon, Dec 23, 2002 at 05:00:29PM -0600 References: <4542F75EC5DC2E44AA0B648E20D00E3504C6EE@rriexmb10.services.reinternal.com> Message-ID: <20021223151554.N29493@shrubbery.net> Mon, Dec 23, 2002 at 05:00:29PM -0600, Zhang, Anchi: > I do get the shell prompt back right after the last line of the debug but it > takes a long time to get there. about 45 seconds? that is the timeout period. but, in the debug output you sent, it should have immediately returned since it matched the prompt and then EOF. however, if the output hung prior to receiving the prompt (when we are not expecting EOF), then it will wait for the timeout period. > Rancid over ssh does work with some of my routers but not all. For > example, it had worked with one router until I added a few more lines to > its existing ACL. can you share the lines that were added? > The strange thing is that if I login using clogin and issue > "show run" at the router's command prompt, the display will be fine. > However, if I do "term len 0" and then "show run" the display hangs when > it gets close to the very end of the config. when clogin is used, clogin takes care of the login process and then uses interact. this should in essence (fingers crossed) connect your terminal directly to the pty until EOF. thus i suspect this is a cisco bug. try telnet (or ssh) without clogin. From AZhang at reliant.com Thu Dec 26 16:33:59 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Thu, 26 Dec 2002 10:33:59 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6EF@rriexmb10.services.reinternal.com> The hang is much longer than 45 seconds: log2% date; clogin -c "sho run" rri2uunet>/dev/null; date Thu Dec 26 10:02:58 CST 2002 Thu Dec 26 10:13:01 CST 2002 I changed the line to "set timeout 10" in clogin but noticed no difference in the hang duration. The lines below added to ACL "ip access-list extended ingress" to make it 181 lines: permit esp any host 158.81.250.11 permit udp any host 158.81.250.11 eq 10000 permit udp any host 158.81.250.11 eq isakmp I can email you the whole ACL in private if you wish to see it. What is even more strange is the fact that I have three edge routers each with the identical ACLs and adding the three lines to the other two routes did not affect Rancid's access to them. ssh or telnet access without clogin presents no problem. clogin via ssh is successful all the time on many routers, once a while on some routers, and never on a few others. Anchi From heas at shrubbery.net Fri Dec 27 17:44:47 2002 From: heas at shrubbery.net (john heasley) Date: Fri, 27 Dec 2002 09:44:47 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E3504C6EF@rriexmb10.services.reinternal.com>; from AZhang@reliant.com on Thu, Dec 26, 2002 at 10:33:59AM -0600 References: <4542F75EC5DC2E44AA0B648E20D00E3504C6EF@rriexmb10.services.reinternal.com> Message-ID: <20021227094447.R12553@shrubbery.net> Thu, Dec 26, 2002 at 10:33:59AM -0600, Zhang, Anchi: > The hang is much longer than 45 seconds: > > log2% date; clogin -c "sho run" rri2uunet>/dev/null; date > Thu Dec 26 10:02:58 CST 2002 > Thu Dec 26 10:13:01 CST 2002 > > I changed the line to "set timeout 10" in clogin but noticed no difference > in the hang duration. see the -t option. > The lines below added to ACL "ip access-list extended ingress" to make it > 181 lines: > > permit esp any host 158.81.250.11 > permit udp any host 158.81.250.11 eq 10000 > permit udp any host 158.81.250.11 eq isakmp > > I can email you the whole ACL in private if you wish to see it. > > What is even more strange is the fact that I have three edge routers each > with the identical ACLs and adding the three lines to the other two routes > did not affect Rancid's access to them. > > ssh or telnet access without clogin presents no problem. clogin via ssh is > successful all the time on many routers, once a while on some routers, and > never on a few others. is it possible that the version of ios running on those suspect routers has a bug related to this ACL? trying reproducing the problem with the ACL removed. otherwise, i'm at a loss. perhaps you can send a successful rancid collection from one of the misbehaving routers to me directly. From AZhang at reliant.com Fri Dec 27 23:31:42 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Fri, 27 Dec 2002 17:31:42 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6F1@rriexmb10.services.reinternal.com> My repeated tests show that the problem is related to the length of the config. The Rancid collection that I mailed you privately was successful because I did UU-Cisco-gw(config)# no ntp server 158.81.250.130 before I ran rancid -d . In fact, shortening the config by just one line, any line, would render Rancid successful. Anchi From heas at shrubbery.net Sun Dec 29 01:48:58 2002 From: heas at shrubbery.net (john heasley) Date: Sat, 28 Dec 2002 17:48:58 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E3504C6F1@rriexmb10.services.reinternal.com>; from AZhang@reliant.com on Fri, Dec 27, 2002 at 05:31:42PM -0600 References: <4542F75EC5DC2E44AA0B648E20D00E3504C6F1@rriexmb10.services.reinternal.com> Message-ID: <20021228174858.E2660@shrubbery.net> Fri, Dec 27, 2002 at 05:31:42PM -0600, Zhang, Anchi: > My repeated tests show that the problem is related to the length of the > config. The Rancid collection that I mailed you privately was successful > because I did > > UU-Cisco-gw(config)# no ntp server 158.81.250.130 > > before I ran rancid -d . In fact, shortening the config by just > one line, any line, would render Rancid successful. hmm, i smell crack. could you try the following on the router: conf t lin v 0 15 exec-time 0 session-time 0 ^Z then try rancid again. From AZhang at reliant.com Mon Dec 30 23:38:25 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Mon, 30 Dec 2002 17:38:25 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6F7@rriexmb10.services.reinternal.com> I tried it and the only difference it seems to have made was to extend the hang period indefinitely. With "exec-timeout 1" the hang is about a minute: log2% date; clogin -c "sho run" ecdc2ibgp>/dev/null; date Mon Dec 30 17:16:17 CST 2002 Mon Dec 30 17:17:20 CST 2002 Today, I was able to have a successful Rancid collection on a router that I had never been able to just by simply removing a few unimportant lines from the config. I was also able to cause a failed collection on a router by simply adding enough lines after ip access-list extended testing Anchi From heas at shrubbery.net Tue Dec 31 00:27:00 2002 From: heas at shrubbery.net (john heasley) Date: Mon, 30 Dec 2002 16:27:00 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E3504C6F7@rriexmb10.services.reinternal.com>; from AZhang@reliant.com on Mon, Dec 30, 2002 at 05:38:25PM -0600 References: <4542F75EC5DC2E44AA0B648E20D00E3504C6F7@rriexmb10.services.reinternal.com> Message-ID: <20021230162700.W17843@shrubbery.net> Mon, Dec 30, 2002 at 05:38:25PM -0600, Zhang, Anchi: > I tried it and the only difference it seems to have made was to extend the hang period indefinitely. > > With "exec-timeout 1" the hang is about a minute: > > log2% date; clogin -c "sho run" ecdc2ibgp>/dev/null; date > Mon Dec 30 17:16:17 CST 2002 > Mon Dec 30 17:17:20 CST 2002 > > Today, I was able to have a successful Rancid collection on a router that I had never been able to just by simply removing a few unimportant lines from the config. I was also able to cause a failed collection on a router by simply adding enough lines after > > ip access-list extended testing could you try the patch for expect that is on www.shrubbery.net/rancid/? From AZhang at reliant.com Tue Dec 31 16:05:33 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Tue, 31 Dec 2002 10:05:33 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E3504C6F8@rriexmb10.services.reinternal.com> Patch just applied but results remain the same. From johan.grip at emea.sykes.com Tue Dec 31 06:53:56 2002 From: johan.grip at emea.sykes.com (Johan Grip) Date: Tue, 31 Dec 2002 06:53:56 -0000 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <59CFF4F7B037D411804800508B6D22B204627A01@UKEDIMAIL02> I am also experiencing this problem as well, and the patch for expect did not help in my case. My setup is cisco routers and a debian 3.0 installation using kernel 2.4.18 and expect 5.32.2. I have for now worked around this by using telnet instead of SSH, but the still remains the same. I am also expriencing this issue sometimes when I run commands with large output without paging on the routers as well, so I do think it might be something in IOS. Currently playing around to see what I can come up with. Regards, Johan From randy at psg.com Tue Dec 31 17:43:32 2002 From: randy at psg.com (Randy Bush) Date: Tue, 31 Dec 2002 09:43:32 -0800 Subject: rancid hangs due to expect, ssh, or cisco? References: <59CFF4F7B037D411804800508B6D22B204627A01@UKEDIMAIL02> Message-ID: > I have for now worked around this by using telnet instead of SSH, but the > still remains the same. I am also expriencing this issue sometimes when I > run commands with large output without paging on the routers as well, > so I do think it might be something in IOS. i think this happens for me on a non-cisco router with a cisco cli randy --- The following routers have not been successfully contacted for more than 4 hours. -rw-r----- 1 randy staff 6928 Nov 18 00:15 psg2.psg.com From asp at partan.com Tue Dec 31 18:14:02 2002 From: asp at partan.com (Andrew Partan) Date: Tue, 31 Dec 2002 13:14:02 -0500 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: References: <59CFF4F7B037D411804800508B6D22B204627A01@UKEDIMAIL02> Message-ID: <20021231181402.GA29656@partan.com> On Tue, Dec 31, 2002 at 09:43:32AM -0800, Randy Bush wrote: > i think this happens for me on a non-cisco router with a cisco cli > --- > The following routers have not been successfully contacted for > more than 4 hours. > -rw-r----- 1 randy staff 6928 Nov 18 00:15 psg2.psg.com That is a failure to log in or connect to the router. A rancid hang would get this message to rancid-admin-$GROUP: Subject: rancid hung - $GROUP echo hourly config diffs failed: $LOCKFILE exists and you would have to go kill -9 some of the rancid programs (typically a hung expect) to get things running again. --asp From heas at shrubbery.net Tue Dec 31 18:16:46 2002 From: heas at shrubbery.net (john heasley) Date: Tue, 31 Dec 2002 10:16:46 -0800 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: ; from randy@psg.com on Tue, Dec 31, 2002 at 09:43:32AM -0800 References: <59CFF4F7B037D411804800508B6D22B204627A01@UKEDIMAIL02> Message-ID: <20021231101646.K17843@shrubbery.net> Tue, Dec 31, 2002 at 09:43:32AM -0800, Randy Bush: > > I have for now worked around this by using telnet instead of SSH, but the > > still remains the same. I am also expriencing this issue sometimes when I > > run commands with large output without paging on the routers as well, > > so I do think it might be something in IOS. > > i think this happens for me on a non-cisco router with a cisco cli almost certainly. if the device_type field of the router.db file is incorrect, the wrong login script might ("might" because some types use the same script) be used. it would be nice to merge all the scripts, but that is difficult - and we dont want to jeopardize their stability. but, i am at a loss as to what might be wrong with Johan or Anchi's collections. i know of rancid users with both solaris and linux that have not had problems. what i've seen thus far point to either an IOS or telnet/ssh/expect problem. someone experiencing the problem will have to figure it out or one of them will have to offer a login and router access to me. sorry guys. From AZhang at reliant.com Tue Dec 31 20:49:48 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Tue, 31 Dec 2002 14:49:48 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E352B0B61@rriexmb10.services.reinternal.com> My temporary workaround to this problem is log2# diff clogin clogin.orig 457c457 < send "term length 100\r" --- > send "term length 0\r" Anchi From asp at partan.com Tue Dec 31 22:07:49 2002 From: asp at partan.com (Andrew Partan) Date: Tue, 31 Dec 2002 17:07:49 -0500 Subject: rancid hangs due to expect, ssh, or cisco? In-Reply-To: <4542F75EC5DC2E44AA0B648E20D00E352B0B61@rriexmb10.services.reinternal.com> References: <4542F75EC5DC2E44AA0B648E20D00E352B0B61@rriexmb10.services.reinternal.com> Message-ID: <20021231220749.GA34124@partan.com> On Tue, Dec 31, 2002 at 02:49:48PM -0600, Zhang, Anchi wrote: > My temporary workaround to this problem is > > log2# diff clogin clogin.orig > 457c457 > < send "term length 100\r" > --- > > send "term length 0\r" Well that is whacko. Is 'term length 0' not working on your router? --asp From AZhang at reliant.com Tue Dec 31 22:15:44 2002 From: AZhang at reliant.com (Zhang, Anchi) Date: Tue, 31 Dec 2002 16:15:44 -0600 Subject: rancid hangs due to expect, ssh, or cisco? Message-ID: <4542F75EC5DC2E44AA0B648E20D00E352B0B63@rriexmb10.services.reinternal.com> Yes, 'term len 0' works on my router. However strange, the workaround does work for me. I was hoping others who have similar problems would try it and confirm. Anchi